I really like Ansible as a quick and accessible automation tool. It is always so low overhead to get going due to being agentless and using YAML. As I've got more experience with config management though, I find myself enjoying other philosophies like Puppet and Chef more. I think once scale gets large, having a consistent execution environment (via an agent) and more expressive language (like Ruby or Puppet DSC over YAML) outweighs the benefits of accessibility Ansible provides.
When I want to get something done quickly and easily though, it's the first thing I reach for.
> Also have a look at community cookbooks, most stuff is very old.
A lot of the functionality offered by community cookbooks made it way into the core DSL. Most community cookbooks for installing applications were abandoned for other approaches though.
>I think chef have totally killed their community by doing that stupid EULA move.
I agree. Seems like they tried to move to the "redhat model" without having a centos style OSS version. CINC has come along and filled that gap: https://cinc.sh/
This is purely a matter of opinion, but in my experience once scale gets large enough where something like Chef or Puppet looks more appealing you really should be looking at containers/pods. It seems to me that Ansible is hitting the right niche between being simple enough to massage the base image for your fleet of docker/kubernetes engines and still works well at a bit of pre and post work for building the containers themselves. It is definitely possible for a good Puppet or Chef system to scale up and handle the whole stack from top to bottom, but I sincerely think that the moment is passing both of those technologies by and the agentless nature of Ansible, its simplicity, and the fact that it leverages Python instead of Ruby is pushing Ansible forward and eclipsing tools that would have been considered competitors five years ago...
This rings true. Systems I manage have gone from using fairly complex configuration management scripts that stand up the full stack, to using really basic scripts that create systemd units to run containers.
Ansible was designed to be agnostic in this regard. Most Ansible plugins in the unix world are written in sh or Python, and in the Windows world in PowerShell. But you can use anything you want.
This is also part of Ansible's philosophy: to be agentless and as thin as possible.
Puppet and chef are significantly more complex, and historically required remote agents on the hosts they managed.
On the threads about Progress buying chef and VMware buying Saltstack, it was pretty clear that the sentiment of the community was that things have moved passed chef and puppet to Ansible and Terraform.
I do agree that Ansible is quite easy to start with, YAML being a fairly simply syntax to understand if you've never coded before.
One of the things i struggle with most in regards to Ansible is that things that are semi-interesting like loops, conditionals, data munging, etc are quite complicated and cumbersome. I agree that things like Puppet and Chef make these tasks _much_ easier and i end up writing more understandable code that works just like any other scripting language.
I also _very_ much appreciate the fact that i can unit test my code in Puppet and Chef. As part of a good DevOps practice, unit testing along with integration/acceptance testing is crucial!
On top of that i believe that the roles in Ansible Galaxy are very much sub-par when compared to modules in Puppet Forge. Most of the modules in Ansible Galaxy are simply "install package, start the service". In general, Puppet Forge modules are much more in-depth and allow for configuration and customization to many aspects of the application you're configuring.
Bolt is an "ad-hoc task runner" very much like Ansible. One of the cool things you can do with Bolt though is apply Puppet code as part of the run, so you can reuse all of the Puppet Forge modules that already exist.
Not sure what does you're using in puppet forge but everyone of the modules I've used are no longer being maintained as the develops have all switched to ansible and left the puppet modules to rot. While ill agree there is a learning curve to ansible for advanced things like loops and conditionals, you have everything a scripting language does.
The important difference between them is not the syntax they are written in, it is their model for how to configure systems.
Ansible is imperative. You give it a list of things to do, and it will do each one, in order, whatever you say.
Puppet is declarative. It runs through your entire script to build a model of how things should be, compares the target system to see how things are different, then uses a combination of implicit and explicit dependencies to order changes and apply them to the machine. Because it is building a model, it is able to detect defects like two different modules trying to configure the same thing two different ways.
The declarative models make it so that doing the right thing is easier then doing the wrong thing (ie, making idempotent configuration is easier then exec'ing out). But the imperative model makes doing everything easier (ie, it doesn't guide me to make idempotent scripts, but makes it super easy to automate any old task list).
That is what makes Ansible easier to adopt, the very low level of mental overhead to understand what it is doing. Conversely, it probably also makes it harder to scale, both because performance is basically single-threaded and synchronous and harder to enforce good standards.
PS I work for Red Hat, but have used both Puppet and Ansible extensively.
I keep trying to get started with puppet/chef, but their homepages are so buried in mounds of enterprise products that it's incredibly hard to figure out how to get started.
Once I eventually figured out which of their indistinct products is actually the basic tool I then tried to follow their tutorials, which want me to install hundreds of megabytes of who knows what, set up heavyweight servers, watch video tutorials (!), and who knows what.
When I eventually slogged through all that to get to a minimally working setup trying to convert a few fairly trivial ansible plays for what I would have thought of as fairly standard stuff rapidly turned into "time to write some ruby"!
I've checked chef/puppet/salt/ansible and all of them full of "contact sales", "request information". I've never seen that much noise on other projects.
I have used and followed Ansible for a long time (iirc I was one of the first hundred people to star it on GitHub in the early days) and while I love certain aspects of it (agentless, strong emphasis on idempotency), I found the yaml format to eventually be limiting and inflexible, and have shifted to a more roll-your-own approach to infra with vanilla python and libraries like paramiko and boto3.
I wish there was a convenient way of using the modules (which are robust and powerful) without the yaml format and boilerplate.
Interesting, that's exactly the reason i like it. If i was working solo, i think i'd have a different view but when working with a team of people, especially teams of people from different groups, i find the enforced common structure liberating.
That said, just to undermine my own point - it is trivial to create an ansible module and i have done that when the alternative YAML would have been horrible.
Have you taken a look at Pulumi? If infrastructure is going to be code, I’d like to code it in a type-safe DSL please. Otherwise refactoring is hell (as in Ansible).
Another promising effort in this direction is Jetbrains Space, which features CI config in a Kotlin DSL.
I'm trying Pulumi at the moment, it seems pretty nice so far - I love that it's not "yaml templates" or some other DSL, but just regular code.
But it seems like it covers a bit of a different space to Ansible... Pulumi will boot up your infrastructure but you still need something like Ansible to install packages on the nodes afterwards.
My impression of Ansible in the past was that it's more complicated than it was supposed to be.
I kind of hope Pulumi will extend in that direction over time, I would rather use one tool for everything and I don't see why it couldn't. The docs for writing your own providers are a bit weak at the moment though.
I recently came across https://pyinfra.com/ which has a similar mental model to Ansible but is just plain Python. For whatever, it's just easier for me to understand the raw Python better than the equivalent YAML.
Personally, I use Ansible for provisioning my Vagrant VM. It's not what it was made for, but I enjoy being able to write the provisioning in a self-documented yaml file using Ansible modules for helpers, instead of writing a shell script to set up the machine.
I just started using Ansible and am quite impressed with its ease of use. We previously did our server setup manually, but we could automate the whole process pretty quickly (I would say 2-3 times the time it costs to set up one deployment manually). The YAML is a bit limiting, as others mention, but to be honest for our use case I'm not sure that we would want to use a more complicated tool instead - there are already more than enough different components and languages in our stack.
I have a lot of different machines performing different tasks, but very few machines running the same tasks as another one, so it's definitely a case of pets rather than cattle for the bulk of work. For the rare occasions a task needs to be done on a variety of machines, clusterssh does perfectly well.
Automation is a preseed, a script to do user management, and a phone home package which sends information about the machine back to a central server.
My main bugbear about ansible is it leaves very little trace -- it runs by copying a file to the remote server in /tmp, executing the file (sometimes under sudo), and then deleting it. I get the notification on syslog that ansible has logged in and done something, but I don't know what, so I rely on the ansible
Compared to when someone manually goes in and does something like "sudo vi /etc/resolv.conf" - even if doing it en-mass, I get notification that joebloggs logged in at 14:53 on Wednesday and edited resolv.conf. If I see something broke about 3pm on Wednesday, that's a great start to working out why it's broken. Ansible means I need to look at the ansible logs, but even then I don't know exactly what was actually executed on the box.
Then of course there's "to foul thing up is ssh, to really foul things up requires ansible" angle too. Automation massively increases the amount of
That may be.. but imagine if you were in a team environment, and others started playing and modifying your wonderful bash code. It's almost a recipe for disaster, unless everyone is as disciplined as you are. With the declarative approach of Ansible, at least most things will be "guaranteed" to work.
I still don't understand why we can't just use bash (or some better language) and reject merge requests that don't have things split up into units that are either pure or who's side effects are idempotent.
Exactly my thinking and what I teach in my book[0]. Things like setting up NGINX or systemd live in separate units and can be rerun (changing NGINX config means just rerunning NGINX part).
One thing this blog doesn’t get into is probably the one thing I struggled with for the longest time. Maybe it’ll come in the next part, but how are people setting up their authentication/access to their hosts? Are you using root and allowing root over ssh? Maybe you’re limiting subnets in sshd_config? Maybe a separate local account named something like “ansible?” With full sudo permissions and nopasswd?
How are people properly hardening this account? I would love to hear how others are doing this because every time I think about our implementation, it just feels wrong.
SSH with public/private key pair or certificate authentication and sudo (optionally with pam_ssh_agent_auth[0]). This of course means you have to have access to the node first before you can do anything with SSH. Which most often is the case if you roll your own base images in cloud environments or use something like cloud-init.
Just curious, what was your rationale for using pam ssh agent auth over nopasswd? It definitely looks more secure as it’d prevent our sudoers from sudoing into our ansible account but that’s about it. We have local console access disabled for the ansible account.
The SSH key you use for sudo can be different than the one you use for accessing the machine, though I never used it that way.
When I first used this setup I tried to build a server configuration that itself was completely open source[0]. So there was no way to bring secrets (like a root or sudo password) to the machine as everything had to come from the source repository so I had to rely on asymmetric authentication.
If you use `become`, you can use an otherwise non-privileged account with sudo rights. When executed with the --ask-become-pass, ansible will interactively prompt you for your sudo password before commencing.
If you feel manually filling in the sudo password every time is cumbersome, you could also integrate it with a yubikey or make it fetch your sudo password automatically via e.g. GNU pass.
For radius do you have a secondary/alternative method of getting into the system in case radius is down? Or is radius a core service for your environment and a P1 issue if it’s ever down?
Experienced the same dilemma. I’ll say what I do and what I want to do.
What I do is SSH as root using SSH certificates. This allows me to use rsync commands in Ansible, because elevating to root after login does not.
What I want to do is generate short-lived SSH certificates for root login. Something like, “Run prod-ssh command, enter password, touch YubiKey, and I can SSH as root for the next four hours.” I know how to make this possible, it would just require a couple days of engineering to do it, and I have other things to do.
If I had more hosts to work with I would make one of them run HashiCorp Vault, and serve as an SSH certificate authority for the others.
If I were a team of people managing servers I would probably switch from Ansible to something else. I would want every configuration action to go through source control before being pushed to live servers. As it is, I’m one person, so I don’t need that.
Evaluate for yourself whether this meets your needs, of course. I want to find my own balance between security, safety (removing foot-guns), and convenience.
For root rsync in Ansible, have you tried using "become: no" and the synchronize module with "rsync_path: sudo rsync"? IIRC that's worked quite well for me.
It'd need passwordless sudo so only a minor improvement, but avoiding root login allows you to limit which tasks run as privileged/non-privileged users, tick boxes when it comes to audits, and allows multiple users to run the same playbooks under their own account.
As you've just said, it might not fit everyone's needs, but thought it's worth putting out there for anybody who would prefer not to log in as root.
If I wanted to tick boxes when it comes to audits, TBH I would migrate from Ansible to something else where changes get pushed to source control first (before push) and nobody has root access during normal operations. I’m using Ansible specifically because I can push directly from my personal machine.
I strongly prefer NOT to have passwordless sudo. Disabling root login and then enabling passwordless sudo seems like a pointless exercise in ticking the boxes—any benefit from disabling root login is undone by enabling passwordless sudo.
Regarding your idea of short lived root SSH certificates. Netflix BLESS[1] is an implementation of this, but it relies on AWS Lambda. Thers's also an open source re-implementation without this dependency, called CURSE[2].
I have a separate playbook called 'provision' that, among other things, creates an unprivileged user, gives it sudo permissions, disables root login and password authentication in sshd_config and then restarts ssh. It's the only playbook that uses root. All subsequent connections are made using the unprivileged user and the become directive for privilege escalation. It's a bit awkward, because the playbook can only be run once, but it works in practice.
I've.. gone a little crazy with it. I have my own Ansible Collection consisting of a common role, a Certificate Authority role, and a pair of SSH Host/Client roles, which together provision and lockdown the CA server, configure the users, principles, and certificates on my machines, and setup a system for future (renewed or revoked) certificate propagation. It was one of my first major ansible projects, and I learned a heck of a lot about best-practice putting it all together. The CA server will renew keys automatically, but can only be accessed with a dedicated key that's left in the playbook directory.
My managed hosts get a special "for Ansible use only" user account (which has "sudo" privileges) created at installation time.
That user is only allowed to log in to those hosts a) with a couple of private keys (i.e., no password authentication allowed) and b) from a few specific (bastion) hosts (controlled via firewall and "Match" rules).
A dedicated private CA and short-lived certificates would be "better" but I haven't yet bothered.
The standard practice is for ops/developers to SSH to servers as themselves. They get sudo privileges on the machines that they should be able to manage.
Usually this is exclusively using SSH keys, password authentication is blocked. root login is blocked too, you can only login as a normal user and sudo.
It's quite easy to manage with ansible itself. Make a configuration file to store public keys and username and that's about it. That will work pretty well until the company has 500 employees and wants to invest into a SSO solution to manage authentication company wide (AD/kerberos/pam).
There's one use case for Ansible that it handles amazingly well that seems to get ignored pretty frequently. It's an extremely useful tool for setting up the same machine it's running on using the local connection type. I've been involved in projects where we setup a small-ish bash wrapper script that can be curled onto a linux machine or mac that installs Ansible, asks a few user specific up front questions, and then installs the rest of a relatively complex development environment. It took a while to build, but makes setting things up so much easier to deal with.
This past year I committed myself to learn ansible and fully script my full desktop setup as a series of ansible roles. This included not just package management but also things like gsettings gnome configuration, gnome extension installs, python virtualenv environments, email / chat client setup, etc.
As it happens, a few months after I did that my hard drive failed on my Centos desktop. I was able to install a new drive, install base OS, and run my ansible role(s). Within ~45 mins I was back to my exact desktop with all local files restored from backup.
Since then I've kept up any small changes by keeping my roles in sync with my setup. The result is a self-documenting setup. I started this as an excuse to learn ansible but I now realize I've relieved lots of mental burdens from myself. Its so nice to go thru life now confident that I can go from complete bare-metal to "my setup" over lunch.
At the moment I'm using git hooks on bare repos on a deployment machine to automatically build detached work trees whenever someone pushes the deployment branch. This works well because no secret keys need exist on the deployment machine. Even if encrypted, I don't want secret keys sitting on a deployment machine.
Does ansible pull require a secret ssh key on the pulling machine? Seems like it would but I don't remember.
It would only require credentials if you need them to fetch from a private git repo (or other protected playbook source); otherwise it (traditionally) uses the "local" connection, meaning it will launch python subprocesses rather than trying to ssh into localhost
I love Ansible, and we migrated our Rails deployment from Chef to Ansible. Now we have a subfolder /ops in our project that contains all the code to provision our infrastructure and deploy our app.
However, I still find it very hard to manage updating roles. Git submodules is just terrible, and you can't have a different git repo for each role subfolder. So how do you manage updating third-party roles?
With Ansible Collections, it'll be easier because it's a proper packaging format. The tarball are uploaded to Galaxy, it's not just pointing to GitHub.
OTOH talking about Git: I usually recommend using subtrees in place of submodules as they allow you to actually merge-in the contents of other trees rather than holding pointers to them.
Roles in collections are still a bit more annoying to deal with (IMO) than roles on their own (which I typically store in individual git repos so I can mix-and-match them in my playbooks more easily, and can test each role in CI thoroughly).
I like ansible as its a great way to run simple deploys in parallel. However as things get more complicated I'm just distributing bash or python scripts to run on each machine. It makes sense to me but I feel I'm doing it wrong.
So ansible "collections" are the next questionable idea in the neverending back-and-forth between "we ship most generally useful modules with ansible itself" and "better luck getting your modules elsewhere, sucker!" that they first tripped down in, if memory serves, 2014 or so? And they, once again, break existing playbook/role syntax? m(
Seriously, it is beyond me how this glorified paralleized shell-loop spewing gzipped python can get away with something like this time and time again... Ansible really is a steaming pile of ill-thought-out ad-hoc decisions, endlessly retrofitted and then retrofitted again, in all tangible qualities and regards. If only we had something better with a big enough audience/"community" that served a similar purpose... sigh
I'm a heavy ansible user and have not experienced anywhere close to this level of pain with ansible 2.10 and collections. None of my playbooks needed module names updated and this is not a syntax change. A fully qualified module name is simply being more explicit about where your desired module comes from in the infinitely extensible library of modules. If you use builtins or have only one of that module name installed you don't have to update anything. At least that has been my experience, although I primarily use builtin modules for a majority of my work.
I maintain a _lot_ of Ansible playbooks, and so far I have not had one of them break after upgrading to Ansible 2.10.
I think in the long term, there may be a point where it will be more important to consider how the new collections are structured if you use ansible in a typical way, but right now at least, it seems like the new release plays nice with all my existing work.
Early in my career, I noticed a mindset among senior developers disparaging ORM's with "I'll write my own SQL, thanks." Although I noted their thoughts at the time, I didn't really do anything with it - there was a mountain of other things to learn and using an ORM made me productive enough to get tasks done.
Of course, years later when I finally dug into SQL and interacting directly with databases, I had a much better understanding of their mindset and wish I would have learned earlier.
Now your mindset feels very similar to me - "I'll write my own shell scripts, thanks." So, to avoid making the same mistake twice, do you have any advice for what to read / learn to better understand your position?
Note: I've written plenty of shell, but the shell I've writen is way to hacky to use as a replacement for ansible.
Shell is not the best solution long-term for a largish organization, but if you are indie hacking a simple app it's actually a viable option.
That's also why I chose Bash to teach config management and deployment[0]. Do it in Bash, understand it fully. Then go a pick up a tool of your liking. Or not, if it actually does it job.
> Early in my career, I noticed a mindset among senior developers disparaging ORM's with "I'll write my own SQL, thanks."
Hopefully they were justified in their choice.
Most senior developers I've worked with who started with this mindset, be it about an ORM, a framework, or some library that they did not understand or wanted to spend some time to learn, ended up just proving that these abstractions were not completely useless after all. How? By recreating their own vastly inferior and headache inducing implementations.
I believe that there's an additional layer of skills between seniority and mastery. The former seems to be just about time and experience, whereas the latter adds a dimension of pragmatism and detachment. You become just a craftsman using tools with indifference to nimbly navigate complexity. You don't blame your tools, you just understand them, strengths and weaknesses. You know when to leverage them and when to ditch them.
I find that with ORMs you'll inevitably run into some kind of corner cases that is not covered, migration issues, library or framework incompatibility. Some argue for query builders but if you're doing that you might as well just write SQL.
There are many things I don't like about Ansible and I don't think it's particularly robust, but it is an improvement over manually written scripts, most of the time, because writing robust scripts is not easy.
What I think "senior developers" are usually disparaging is using tools like ORMs without appropriate understanding of how they work, and thus being unable to recognize when they don't suit the task at hand.
"glorified paralleized shell-loop spewing gzipped python can get away with something like this time and time again"
That's exactly why it's good. It beat puppet, it beat chef, and saltstack fails to make inroads for the same reasons. They all make useless and difficult abstraction layers on top of what is essentially running shell scripts in loops over what's ultimately an ssh connection or a worse, proprietary version of one.
I won't defend some of the decisions Ansible has made over the years, but its simplicity is the key feature.
I started writing one about the time Ansible debuted because of some of its design decisions that I disliked. A bit later Docker dropped and I decided most of the whole category was approaching obsolescence already and stopped.
Given the above, and that a competitor would have to overcome Red Hat/investor money and mindshare, it doesn't seem there would be much of a demand for this. Ansible is arguably good enough for the niche it occupies.
Recently I give it 2nd try thinking that perhaps I'm being too perfectionist and close-minded. In about 1h I realized that no, it's not me. Ansible is really just a huge masturbation. You'd think that a tool that purportedly helps with managing servers would at least give you meaningful error messages but it still spews JSONs, that don't even start and end on its own lines, where actual error message is single line with \n for line breaks.
I have also tried cdist[1]. Compared to glorified shell command generator like Ansible, cdist is actually closer to Puppet in that you actually declare resources and dependencies, where by "declare" I don't mean that there's a guard in imperative steps like in Ansible.
Ansible is nice but I like to build my systems the way I like them. Under the hood ansible is just a Python paramiko wrapper anyway. What I like about bash is it’s just there, no need to install anything if you are managing legacy systems
It doesn’t work if the step blocks, needs some error handling, etc. Then comes the fun of writing yet another deployment system that fails to learn from other systems.
Why we keep reinventing wheels in infrastructure management and control beats me
One of Ansible's big selling points that it was "simple" because it used YAML.
What a shitty idea that turned out to be.
And it's not simple. It's just as complicated as its competitors. It's full of weird quirks and corner cases, and just like all the other tools in its class it feels like it was more congealed and cobbled together out of random bits than designed in a consistent, elegant way.
The most promising thing in this area, IMO, is NixOS and surrounding tools like NixOps. They're strange and hard to learn, and you have to do almost everything differently from what you learned, but it is really true declarative configuration of a machine from the ground up, not quasi-declarative where there's always state that gets left out and you can never really trust the diffing/idempotency mechanisms because they're layered on top of things that weren't meant to be used that way.
(If you dig into the details, you will find some "atrocious hacks" as the sibling comment puts it, but because of how it works, the hacks are almost completely encapsulated at build time and those builds are reproducible, so there's much less to worry about when you're deploying/provisioning.)
Anyone who thinks that any tool will make systems administration and automation "simple" has drank the kool-aide of some sales pitch.
Systems administration and automation is complex.
That being said I've found Ansible to be the most useful tool for the job. It does the things I need it to and new modules are usually easy to incorporate into my work if I need specific tasks done. It has its shortcomings but it hasn't blown up into an unmaintainable mess like many of the other solutions I've poured hundreds of hours into.
YAML is not fun, but it's been good enough for the task of pushing parameters around.
If you don't try throwing tons of logic in the YAML (and build a module or plugin where needed to keep that logic out of the YAML), then your playbooks are 99% of the time simple YAML lists and dicts.
The hardest thing for some people seems to be whitespace issues, but that can be resolved easily enough by integrating a linter into your environment, or minimally showing whitespace characters in your editor.
> then your playbooks are 99% of the time simple YAML lists and dicts.
But really, that's not the problem. The problem is when you inherit someone else's playbook that is itself trying to do a hackjob around a community playbook.
When you inherit any code, it's always a mess, at least that's my experience. Sometimes a working mess, but it's rare to inherit nice code in any system.
I could be biased since I've worked mostly with Python, Ruby, PHP, and Java.
of course, but debugging ansible YAML is exceptionally hard on account of the difficulty of decoding its quasi-declarative, quasi-imperative nature, the difficulty of knowing what the variables to be interpolated are (they could have been changed by other playbooks!), etc. You'll want at the very least a sane set of debug tools to figure things out when they go wrong.
The other alternative is to just give up and rewrite the ansible script, which has been what I had to do on several occasions.
Whitespace issues plague careless people in general, but ansible's choice to run jinja2 using its default delimiters of `{` was the million dollar mistake since it forces quoting to start a scalar with that character and it's quoting issues in yaml which compromise a significant portion of the SO questions I triage
Even allowing a playbook to opt out of the default delimiters in the same way one can in a template file[1] would allow a transition out of that mess
As a long time puppet user, I started forcing myself to use ansible for my homelab so I can learn it's intricacies and limitations.
Recently I had a use-case where I wanted to re-use some data from a defined variable (list of objects) in a second module, but the format had to be different. In python I would have done something like new_var=[{a:i['a'], b:i['b']} for i in old_var], but apparently that is impossible in ansible. The best solution I was able to find after lots of searching was to to pass new_var to a jinja2 template which would render new_var as text, then run that through a json interpreter and set that as new_var. Absolutely rediculous.
It astounds me that a tool BUILT on python lacks any semblance of python's biggest feature, list comprehension.
Modules are where the python lives. Really well written, idempotent, and battle tested python.
But "writing" Ansible playbooks is definitely not python. It's its own dsl bastardized between yaml, jinja2 and reserved/special keywords. This is the worst thing I can say about the whole toolkit.
Unfortunately, I'm really productive with it and when I return to work I've written 3+ years ago:
A) It still runs
2) I can read and understand the intent of my playbooks
Tangential question for people who've used both Ansible and Chef. I have most of our IaaS code in a decently large Chef repo. However it seems like that product is slowly circling the drain, and everyone's converging on Ansible. Should I be migrating over to Ansible now?
It depends on your infra. If you're largely ephemeral hosts, then Ansible is ideal for single instance provisioning. Also if you're support windows hosts Ansible's support here is less than stellar, it's effectively a execute wrapper for running powershell scripts you have to write.
If your hosts have a long duty-cycle Chef may still work better as it handles upgrades and MIA hosts coming back a bit better. I believe someone else in the comments mentioned the OSS project for it https//cinc.sh if you go that route. I've generally found Chef to scale better beyond 1000 nodes, up to about 100k.
I love Ansible, but I'm waiting for the day that reproducible builds are part of most distributions. Nix is paving the way, and it seems like the rest are following. Then maybe the need for these configuration management tools will lessen. Look here for more info: https://reproducible-builds.org/
One of the useful parts of Ansible was its windows modules. I know that Microsoft has made DSC (Desired State Configuration) to handle config management in a declarative way, but haven't tried it out.
Although I don't 'love' Ansible I use it nearly daily and have written thousands of lines of Ansible YAML. It suits, but yes, I share your view; this stuff should be unnecessary. Docker as well.
Docker is a fairly precise proxy for the inadequacies of Linux; all it's really doing is papering over the missing, badly needed parts of the OS.
So what's the hot take on using docker-compose vs. something like Ansible? Anyone have a list of interesting pros/cons to consider before spinning up a next project on one or the other?
Using containers + docker-compose in my own personal projects over the last few months has changed my life, so always interested in learning some more granular info re: best practices and potential pitfalls... Thanks!
It seems to me that Ansible is best for deploying servers, and containers are best for deploying applications.
You can absolutely use Ansible to deploy applications, but, if there is any essential complexity in your run-time environment or configuration needs, it seems that Ansible has a tendency to make that complexity compound multiplicatively instead of additively.
Disclaimer: I say this, not as an ops person, but as a developer who often uses containers and occasionally watches people wrestling with Ansible from the sidelines.
I often use both—docker-compose (or something more heavyweight like k3s) to manage my apps and deployments, but Ansible to make sure docker / k3s / security stuff is configured correctly on the servers themselves.
102 comments
[ 3.3 ms ] story [ 160 ms ] threadWhen I want to get something done quickly and easily though, it's the first thing I reach for.
You can unit test your configuration (which in ansible isn't really practical due to its dynamic nature). You can get completion in your editor...
In practice, though, you write a lot more code and have to comprehend more. Also have a look at community cookbooks, most stuff is very old.
I think chef have totally killed their community by doing that stupid EULA move. https://docs.chef.io/versions/
A lot of the functionality offered by community cookbooks made it way into the core DSL. Most community cookbooks for installing applications were abandoned for other approaches though.
>I think chef have totally killed their community by doing that stupid EULA move.
I agree. Seems like they tried to move to the "redhat model" without having a centos style OSS version. CINC has come along and filled that gap: https://cinc.sh/
This is also part of Ansible's philosophy: to be agentless and as thin as possible.
Puppet and chef are significantly more complex, and historically required remote agents on the hosts they managed.
On the threads about Progress buying chef and VMware buying Saltstack, it was pretty clear that the sentiment of the community was that things have moved passed chef and puppet to Ansible and Terraform.
One of the things i struggle with most in regards to Ansible is that things that are semi-interesting like loops, conditionals, data munging, etc are quite complicated and cumbersome. I agree that things like Puppet and Chef make these tasks _much_ easier and i end up writing more understandable code that works just like any other scripting language.
I also _very_ much appreciate the fact that i can unit test my code in Puppet and Chef. As part of a good DevOps practice, unit testing along with integration/acceptance testing is crucial!
On top of that i believe that the roles in Ansible Galaxy are very much sub-par when compared to modules in Puppet Forge. Most of the modules in Ansible Galaxy are simply "install package, start the service". In general, Puppet Forge modules are much more in-depth and allow for configuration and customization to many aspects of the application you're configuring.
Also, check out Bolt from Puppet: https://puppet.com/docs/bolt/latest/bolt.html
Bolt is an "ad-hoc task runner" very much like Ansible. One of the cool things you can do with Bolt though is apply Puppet code as part of the run, so you can reuse all of the Puppet Forge modules that already exist.
Ansible is imperative. You give it a list of things to do, and it will do each one, in order, whatever you say.
Puppet is declarative. It runs through your entire script to build a model of how things should be, compares the target system to see how things are different, then uses a combination of implicit and explicit dependencies to order changes and apply them to the machine. Because it is building a model, it is able to detect defects like two different modules trying to configure the same thing two different ways.
The declarative models make it so that doing the right thing is easier then doing the wrong thing (ie, making idempotent configuration is easier then exec'ing out). But the imperative model makes doing everything easier (ie, it doesn't guide me to make idempotent scripts, but makes it super easy to automate any old task list).
That is what makes Ansible easier to adopt, the very low level of mental overhead to understand what it is doing. Conversely, it probably also makes it harder to scale, both because performance is basically single-threaded and synchronous and harder to enforce good standards.
PS I work for Red Hat, but have used both Puppet and Ansible extensively.
Once I eventually figured out which of their indistinct products is actually the basic tool I then tried to follow their tutorials, which want me to install hundreds of megabytes of who knows what, set up heavyweight servers, watch video tutorials (!), and who knows what.
When I eventually slogged through all that to get to a minimally working setup trying to convert a few fairly trivial ansible plays for what I would have thought of as fairly standard stuff rapidly turned into "time to write some ruby"!
I gave up.
I wish there was a convenient way of using the modules (which are robust and powerful) without the yaml format and boilerplate.
That said, just to undermine my own point - it is trivial to create an ansible module and i have done that when the alternative YAML would have been horrible.
Another promising effort in this direction is Jetbrains Space, which features CI config in a Kotlin DSL.
But it seems like it covers a bit of a different space to Ansible... Pulumi will boot up your infrastructure but you still need something like Ansible to install packages on the nodes afterwards.
My impression of Ansible in the past was that it's more complicated than it was supposed to be.
I kind of hope Pulumi will extend in that direction over time, I would rather use one tool for everything and I don't see why it couldn't. The docs for writing your own providers are a bit weak at the moment though.
https://pyinfra.com/
https://github.com/Fizzadar/pyinfra
HN discussion: https://news.ycombinator.com/item?id=23487178
I always describe it as "bash on steroid", it's super easy to write and to run remotely on other servers.
Automation is a preseed, a script to do user management, and a phone home package which sends information about the machine back to a central server.
My main bugbear about ansible is it leaves very little trace -- it runs by copying a file to the remote server in /tmp, executing the file (sometimes under sudo), and then deleting it. I get the notification on syslog that ansible has logged in and done something, but I don't know what, so I rely on the ansible
Compared to when someone manually goes in and does something like "sudo vi /etc/resolv.conf" - even if doing it en-mass, I get notification that joebloggs logged in at 14:53 on Wednesday and edited resolv.conf. If I see something broke about 3pm on Wednesday, that's a great start to working out why it's broken. Ansible means I need to look at the ansible logs, but even then I don't know exactly what was actually executed on the box.
Then of course there's "to foul thing up is ssh, to really foul things up requires ansible" angle too. Automation massively increases the amount of
Code review is the key in both cases.
[0] https://deploymentfromscratch.com/
How are people properly hardening this account? I would love to hear how others are doing this because every time I think about our implementation, it just feels wrong.
[0] http://pamsshagentauth.sourceforge.net/
When I first used this setup I tried to build a server configuration that itself was completely open source[0]. So there was no way to bring secrets (like a root or sudo password) to the machine as everything had to come from the source repository so I had to rely on asymmetric authentication.
[0] https://gitlab.com/internet-cleanup-foundation/server
If you feel manually filling in the sudo password every time is cumbersome, you could also integrate it with a yubikey or make it fetch your sudo password automatically via e.g. GNU pass.
What I do is SSH as root using SSH certificates. This allows me to use rsync commands in Ansible, because elevating to root after login does not.
What I want to do is generate short-lived SSH certificates for root login. Something like, “Run prod-ssh command, enter password, touch YubiKey, and I can SSH as root for the next four hours.” I know how to make this possible, it would just require a couple days of engineering to do it, and I have other things to do.
If I had more hosts to work with I would make one of them run HashiCorp Vault, and serve as an SSH certificate authority for the others.
If I were a team of people managing servers I would probably switch from Ansible to something else. I would want every configuration action to go through source control before being pushed to live servers. As it is, I’m one person, so I don’t need that.
Evaluate for yourself whether this meets your needs, of course. I want to find my own balance between security, safety (removing foot-guns), and convenience.
It'd need passwordless sudo so only a minor improvement, but avoiding root login allows you to limit which tasks run as privileged/non-privileged users, tick boxes when it comes to audits, and allows multiple users to run the same playbooks under their own account.
As you've just said, it might not fit everyone's needs, but thought it's worth putting out there for anybody who would prefer not to log in as root.
I strongly prefer NOT to have passwordless sudo. Disabling root login and then enabling passwordless sudo seems like a pointless exercise in ticking the boxes—any benefit from disabling root login is undone by enabling passwordless sudo.
[1]: https://github.com/Netflix/bless
[2]: https://github.com/mikesmitty/curse
* Prompt for username and password
* Create user
* Add it to the wheel group
* Copy over authorized keys
* Edit sshd_config with:
* Restart sshdThat user is only allowed to log in to those hosts a) with a couple of private keys (i.e., no password authentication allowed) and b) from a few specific (bastion) hosts (controlled via firewall and "Match" rules).
A dedicated private CA and short-lived certificates would be "better" but I haven't yet bothered.
Usually this is exclusively using SSH keys, password authentication is blocked. root login is blocked too, you can only login as a normal user and sudo.
It's quite easy to manage with ansible itself. Make a configuration file to store public keys and username and that's about it. That will work pretty well until the company has 500 employees and wants to invest into a SSO solution to manage authentication company wide (AD/kerberos/pam).
As it happens, a few months after I did that my hard drive failed on my Centos desktop. I was able to install a new drive, install base OS, and run my ansible role(s). Within ~45 mins I was back to my exact desktop with all local files restored from backup.
Since then I've kept up any small changes by keeping my roles in sync with my setup. The result is a self-documenting setup. I started this as an excuse to learn ansible but I now realize I've relieved lots of mental burdens from myself. Its so nice to go thru life now confident that I can go from complete bare-metal to "my setup" over lunch.
In your scenario, once those changes are captured, they can be written into `/etc/ansible/facts.d` and they'll show up in future playbook runs: https://docs.ansible.com/ansible/2.10/user_guide/playbooks_v...
Does ansible pull require a secret ssh key on the pulling machine? Seems like it would but I don't remember.
However, I still find it very hard to manage updating roles. Git submodules is just terrible, and you can't have a different git repo for each role subfolder. So how do you manage updating third-party roles?
OTOH talking about Git: I usually recommend using subtrees in place of submodules as they allow you to actually merge-in the contents of other trees rather than holding pointers to them.
Seriously, it is beyond me how this glorified paralleized shell-loop spewing gzipped python can get away with something like this time and time again... Ansible really is a steaming pile of ill-thought-out ad-hoc decisions, endlessly retrofitted and then retrofitted again, in all tangible qualities and regards. If only we had something better with a big enough audience/"community" that served a similar purpose... sigh
I think in the long term, there may be a point where it will be more important to consider how the new collections are structured if you use ansible in a typical way, but right now at least, it seems like the new release plays nice with all my existing work.
Of course, years later when I finally dug into SQL and interacting directly with databases, I had a much better understanding of their mindset and wish I would have learned earlier.
Now your mindset feels very similar to me - "I'll write my own shell scripts, thanks." So, to avoid making the same mistake twice, do you have any advice for what to read / learn to better understand your position?
Note: I've written plenty of shell, but the shell I've writen is way to hacky to use as a replacement for ansible.
You obviously haven't looked at the code for some of the Ansible modules!
That's also why I chose Bash to teach config management and deployment[0]. Do it in Bash, understand it fully. Then go a pick up a tool of your liking. Or not, if it actually does it job.
[0] https://deploymentfromscratch.com/
Hopefully they were justified in their choice.
Most senior developers I've worked with who started with this mindset, be it about an ORM, a framework, or some library that they did not understand or wanted to spend some time to learn, ended up just proving that these abstractions were not completely useless after all. How? By recreating their own vastly inferior and headache inducing implementations.
I believe that there's an additional layer of skills between seniority and mastery. The former seems to be just about time and experience, whereas the latter adds a dimension of pragmatism and detachment. You become just a craftsman using tools with indifference to nimbly navigate complexity. You don't blame your tools, you just understand them, strengths and weaknesses. You know when to leverage them and when to ditch them.
There are many things I don't like about Ansible and I don't think it's particularly robust, but it is an improvement over manually written scripts, most of the time, because writing robust scripts is not easy.
What I think "senior developers" are usually disparaging is using tools like ORMs without appropriate understanding of how they work, and thus being unable to recognize when they don't suit the task at hand.
The ideal tool is good in the general case, and still reliable in the edge case.
That's exactly why it's good. It beat puppet, it beat chef, and saltstack fails to make inroads for the same reasons. They all make useless and difficult abstraction layers on top of what is essentially running shell scripts in loops over what's ultimately an ssh connection or a worse, proprietary version of one.
I won't defend some of the decisions Ansible has made over the years, but its simplicity is the key feature.
I started writing one about the time Ansible debuted because of some of its design decisions that I disliked. A bit later Docker dropped and I decided most of the whole category was approaching obsolescence already and stopped.
Given the above, and that a competitor would have to overcome Red Hat/investor money and mindshare, it doesn't seem there would be much of a demand for this. Ansible is arguably good enough for the niche it occupies.
Recently I give it 2nd try thinking that perhaps I'm being too perfectionist and close-minded. In about 1h I realized that no, it's not me. Ansible is really just a huge masturbation. You'd think that a tool that purportedly helps with managing servers would at least give you meaningful error messages but it still spews JSONs, that don't even start and end on its own lines, where actual error message is single line with \n for line breaks.
I have also tried cdist[1]. Compared to glorified shell command generator like Ansible, cdist is actually closer to Puppet in that you actually declare resources and dependencies, where by "declare" I don't mean that there's a guard in imperative steps like in Ansible.
[1] https://www.cdi.st/
for i in $(cat listofips);do ssh $i “some linuxy thing” ;done
Why we keep reinventing wheels in infrastructure management and control beats me
What a shitty idea that turned out to be.
And it's not simple. It's just as complicated as its competitors. It's full of weird quirks and corner cases, and just like all the other tools in its class it feels like it was more congealed and cobbled together out of random bits than designed in a consistent, elegant way.
(If you dig into the details, you will find some "atrocious hacks" as the sibling comment puts it, but because of how it works, the hacks are almost completely encapsulated at build time and those builds are reproducible, so there's much less to worry about when you're deploying/provisioning.)
Systems administration and automation is complex.
That being said I've found Ansible to be the most useful tool for the job. It does the things I need it to and new modules are usually easy to incorporate into my work if I need specific tasks done. It has its shortcomings but it hasn't blown up into an unmaintainable mess like many of the other solutions I've poured hundreds of hours into.
YAML is not fun, but it's been good enough for the task of pushing parameters around.
The hardest thing for some people seems to be whitespace issues, but that can be resolved easily enough by integrating a linter into your environment, or minimally showing whitespace characters in your editor.
But really, that's not the problem. The problem is when you inherit someone else's playbook that is itself trying to do a hackjob around a community playbook.
I could be biased since I've worked mostly with Python, Ruby, PHP, and Java.
The other alternative is to just give up and rewrite the ansible script, which has been what I had to do on several occasions.
Even allowing a playbook to opt out of the default delimiters in the same way one can in a template file[1] would allow a transition out of that mess
1 = https://docs.ansible.com/ansible/2.10/collections/ansible/bu...Recently I had a use-case where I wanted to re-use some data from a defined variable (list of objects) in a second module, but the format had to be different. In python I would have done something like new_var=[{a:i['a'], b:i['b']} for i in old_var], but apparently that is impossible in ansible. The best solution I was able to find after lots of searching was to to pass new_var to a jinja2 template which would render new_var as text, then run that through a json interpreter and set that as new_var. Absolutely rediculous.
It astounds me that a tool BUILT on python lacks any semblance of python's biggest feature, list comprehension.
But "writing" Ansible playbooks is definitely not python. It's its own dsl bastardized between yaml, jinja2 and reserved/special keywords. This is the worst thing I can say about the whole toolkit.
Unfortunately, I'm really productive with it and when I return to work I've written 3+ years ago:
A) It still runs
2) I can read and understand the intent of my playbooks
If your hosts have a long duty-cycle Chef may still work better as it handles upgrades and MIA hosts coming back a bit better. I believe someone else in the comments mentioned the OSS project for it https//cinc.sh if you go that route. I've generally found Chef to scale better beyond 1000 nodes, up to about 100k.
One of the useful parts of Ansible was its windows modules. I know that Microsoft has made DSC (Desired State Configuration) to handle config management in a declarative way, but haven't tried it out.
Docker is a fairly precise proxy for the inadequacies of Linux; all it's really doing is papering over the missing, badly needed parts of the OS.
Using containers + docker-compose in my own personal projects over the last few months has changed my life, so always interested in learning some more granular info re: best practices and potential pitfalls... Thanks!
You can absolutely use Ansible to deploy applications, but, if there is any essential complexity in your run-time environment or configuration needs, it seems that Ansible has a tendency to make that complexity compound multiplicatively instead of additively.
Disclaimer: I say this, not as an ops person, but as a developer who often uses containers and occasionally watches people wrestling with Ansible from the sidelines.