One of the things that got me into ansible was simply doing
ansible all -i ./ansible_hosts -m command -a "cat /etc/issue"
From there I started going trough the possible options I could pass instead of each time doing a command, such as checking the network speed using a script etc..
As it might be useful for some I have next aliases set:
export ANSIBLE_PATH="$HOME/web/clusters-servers/mycluster"
alias a="ansible -i $ANSIBLE_PATH/hosts"
alias ap="ansible-playbook -i $ANSIBLE_PATH/hosts"
# for vagrant testing machines where I test playbooks before deploying
alias atest="ansible -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s"
alias aptest="ansible-playbook -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s"
On of the (slightly) annoying things is after bringing up a new node you have to "bootstrap" the node so you can issue ansible commands on it, I have a simple script [1] for doing so but I'd love to hear how others do it!
The raw command. Doesn't require anything to be installed on the target host, so you can set up a common playbook which will bootstrap the rest of Ansible.
For a more low-tech solution, I've found cssh / csshX ("Cluster SSH") quite the timesaver. It pops up a bunch of xterms/Terminal.app windows and lets you type in one window while sending the keystrokes to all of the ssh sessions simultaneously.
The problem with cluster SSH is it's not declarative, so if one server already has a package, and another does not, your servers will not behave the same if typing into all of them.
This is why Ansible modules (and most other config tools) instead describe the desired end state, and perform the commands needed to get you to those states.
Basically all config tools arose out of a need for solutions to replace parallel scripting with something more robust that accounted for potential variance between systems, and also to make things more push-button.
I can see that for some kinds of admin tasks, but why package management? At least on Debian, the package manager is already declarative: There's an install/upgrade planner that figures out how to get from the current system state to the final desired state, given constraints such as packages' declared Requires/Conflicts/etc. lines. If the system is already at that state, it does nothing.
How often is setting up a package just running apt-get? There is an overwhelming majority of times where you have to change something in the config file somewhere. Ansible can do anything you want to provision your boxes.
Indeed, cssh is great for getting a quick visual on a few servers, but you're pretty much limited to what will fit on your screen. Ansible really shines when you scale it up a bit further, and it grows well, allowing you to add bits and pieces as needed.
The hosts/inventory in ansible as a source of truth for your infrastructure is also a nice feature. You could use a selection from your ansible inventory to open a cssh with 'top' running on all your webservers, for example.
For testing ansible playbooks/roles in a vagrant VM, I find a simpler way is to use Vagrant's built-in support for Ansible as a provisioner.
This way, launching your test VM is just a `vagrant up` and running ansible against it (repeatedly, if needed) is `vagrant provision`. No need to pass inventory/key parameters at the shell this way - Vagrant calls Ansible with the right settings.
One thing that's easy to miss is to make Ansible use sudo when running against the Vagrant VM, since the vagrant user by convention has passwordless sudo rights (ansible.sudo = true in the Vagrantfile).
Or, if you run more than one project with multiple servers each, ansible will also read an ansible.config file in the current directory. You can have a separate hosts list, options, and playbooks for each set of servers. Then just cd ~/projects/whatever and ansible will work accordingly.
You can put any given set of playbooks and configs into git and share with your team so that everyone is using ansible the same way. And that repo won't even carry any auth info.
For Cloud services, SSH is "cloud native" meaning authorized keys are automatically injected by the cloud provider.
If using metal installs, you can also set this up in your preseed or kickstart.
However, you should never have to bootstrap a machine with another script to install things like python-apt, as you can just use the "raw" command to do this for anything that is not part of the box, see http://docs.ansible.com/raw_module.html - so it's still self-bootstrapping.
Thanks! Didn't knew about this, or just read over it. I'm mostly using ansible for learning how to use it, don't have huge clusters yet to manage. Most of the hosts I have come from LEB (LowEndBox) offerings so therefor no way to deploy my own base image.
And thank you for the work you put into ansible sir!
I've long been accustomed to setting up servers one-command-at-a-time with SSH. I tried Chef and some of the others, but they seemed too complicated to get setup.
Ansible's out of the box experience allows you to start with some simple but useful constructs then allow you to graduate to more and more complex usage.
I started with some of the one-liners here, then setting up inventory files, then roles, some simple tasks and then (what really sold me) was templates.
Even just having those configuration files in an SCM is a huge win, but I honestly squealed with joy when I was able to write a nginx configuration file that dynamically included backend servers given a particular role. And that provisioning a new backend server took a couple minutes and was automatically added to the load-balancing proxy! (not to mention was instantly sending metrics to a graphite/statsd backend because collectd was installed automatically too as part of a common role and configured to point to that log collecting server!)
Ansible is great, but could really have a better first time experience. The system itself is incredibly simple, but the docs and instructions are anything but, and seem to be modeled after Chef and Puppets which are frankly terrible. Instead, look at something like Vagrant as the shining example of how to do it right.
Just compare http://www.ansible.com/ with the excellent http://www.vagrantup.com/ when it comes to downloading and running your first commands. After clicking around for a good while on the Ansbile site you finally find "To install on a Mac, make sure you have Homebrew, then run... ", yeah, no thanks. Oh, there's a "pip" command, if I happen to have that installed (hint: nobody on Mac has by default), but at least there's some more magic commands I can run to install that first. After running the "Looks Totally Safe To Me! (TM)" command "sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install ansible" it blows up on Mavericks with some error about something called paramiko and headers not being available. The attrition rate for newbies on Mac must be phenomenal, because you have to be really dedicated to give it another shot after this. What I'm getting at is, Python software distribution is an abysmal experience, always has been, and you should not subject your users to it. They're not Python developers, they're largely not even devs at all, they don't care about your pip's and brews.
Compare this with Vagrant: Double click this here DMG, run "vagrant init", "vagrant up" and done, in nice _big_ friendly letters on the web site.
Disclaimer: I use and love Ansible (on Linux), and want to see it succeed. Telling my Mac using ops team to check it out soured everyone's first impression right from the start. Rant over.
As someone brand new to Ansible, and having been developing on OSX for years now, I'd just like to play devil's advocate.
Finding the Ansible docs (http://docs.ansible.com/) was not as easy as it could be, however, the install instructions were easily accessible from there, exactly where I'd expect them: Introduction > Installation > Latest Releases via Homebrew.
Second, yeah, pip sucks, I try to avoid it at all costs. However, homebrew (http://brew.sh/) is that one thing that all OSX power users should install as soon as they touch their workstation. 90% of the software - CLI or GUI - on my MacBook is installed and managed by homebrew. I cannot recommend it highly enough. In fact, I didn't even install Ansible through their instructions. I read about it somewhere else, and thought, "Hmmm, I wonder if this is in homebrew? `brew search ansible` Oh! Sweet! `brew install ansible`" Worked perfectly the first time, and I haven't fiddled with it since.
Note, I wrote the program, but I'd disagree very very strongly with this - we have a giant ton of Mac users.
I'm sorry you had install problems and had trouble reading our install chapter: http://docs.ansible.com/intro_installation.html ... I will say the docs are open source, so if you want to add a note about how to get something easier compiled on OS X or something, click the GitHub link in the upper right and you're off and running :)
We get a lot of strong positives about how awesome our docs are, so I'm really interested in what you think we could do to improve them -- but they were absolutely not modelled on Chef and Puppets. Chef has about 1000 pages of docs, Ansible covers the majority of everything in a dozen pages. Most folks get up and running (with working playbooks running real applications) in 30 minutes or an hour.
My philosophy on docs are mainly to address things in the order folks will want to learn things, but also make it easy to hop over sections people will not. Show examples, don't make a reference manual. Though there's a need for reference content, which is where stuff like galaxy.ansible.com content and github.com/ansible/ansible-examples steps in, so you can see it put together. More tutorials I think would still be useful in the main docs. But yeah, a discussion on what you like to see differently would be interesting.
While I do understand you have to do something to install the program, what you have to do to configure it once installed is pretty short, and the install itself should still only take a few minutes. I'll give you the fact that Macs are not Linux, but they are getting there :)
I'm interested in your Mavericks error and you should definitely stop by the list about that one, or just consider the homebrew install.
Another great thing to do since you are already are on a Mac is just start doing your development on Linux in a VM :) Not that we require it, we don't... but it's great when you've got a real package manager stock!
Another easy thing you can do is just checkout the git source and do "source ./hacking/env-setup" and then you are running straight out of source. This is what I do on OS X most of the time, and we have a large number of users that follow the development branch full time.
Thank you for the reply! It's great to see devs interacting on HN. I've written thousands of lines of Ansible playbooks, and I'm a huge fan. I see others on my team struggling with these very basic on-boarding issues, that I think aren't perhaps obvious when you're so embedded in the Linux mindset.
You should absolutely think of your software, even though it's free, as having a conversion funnel to happy users. You are competing both against established players like Chef and Puppet, as well as new up and comers like Salt. Once a potential user reaches ansible.com, at every single step of the on-boarding that follows you will have some % of attrition to your competitors and apathy. The more steps, the larger the attrition, the smaller your userbase is.
Think of it this way; if you add a new feature, you may gain some small percentage of users. But if you reduce the on-boarding attrition rate from 30% to, 1-2%, you will gain a simply massive amount of users over time. Do you know or track your conversion rate?
The best docs is no docs. When curious Mac user visits http://www.vagrantup.com/ they see the download button, they click the Mac installer, and they're done. At no point during the on-boarding does installing the software require even seeing the docs. Mac users and even devs expect DMGs these days. Everything from 'git', to 'vagrant', come in DMGs. On most sites (like git), you visit the front page and it autodetects you are on Mac and gives you the DMG, which every Mac user will immediately know how to use. Git comes installed by default in Macs these days. Most people on Mac never need to install git anymore. The DMG file for git on Mac has 11,000+ weekly downloads on git-scm.com (that's in addition to the brews and ports of the world, which I would imagine have less users). How would you like an extra 11K downloads a week? ;)
The tide has definitely turned in favor of Macs over the last 3 years in software shops, I see more and more developers using Macs exclusively for development. In my company, although all software gets deployed to Linux (on AWS), of the several hundred devs working on the software I would be surprised if more than a handful (like me) still use Linux on their development machines (deploying to Vagrant locally and the like). While many of the Mac converts are coming from Linux, most new devs are just given a MacBook Pro from day one. In this new reality, I fear that you are doing yourself a big disservice by trying to shoehorn the Linux software distribution models on the new Mac users.
Well I would like to add my 2 cents and I have to say my experience with Ansible has been comparably bad on Windows as a development machine. I have my tools in Windows, I rather prefer not having to duplicate all that in order to do some devops.
People used to tell it is possible to bootstrap Ansible via Vagrant on Windows:
1. define a linux box as ansible control vm, provision it using vagrant's shell provider
2. mount the playbook and run it
Well, I have spent hours on this. There were errors with Python in Linux (I forgot which, but I could solve this via searches in news groups), mounting rights, but most importantly I couldn't get the ssh part working in a multi-machine setup.
I still don't understand how this could happen without Ansible halting, but Ansible did apply all the remote playbooks on the control vm itself. Even whith explicitly different ip's specified.
In the end I did gave up, having wasted so much time on a productivity tool.
I am ready to accept this is due to me being stupid, but other tools don't make me look as such which is better for my self esteem.
I'll have a new try soon as Vagrant and Ansible keeps maturing and maybe I can find a tutorial this time which works with the current state of Ansible.
I think that it's curious how you keep referring to Vagrant as an example of "All That is Good And Proper(tm)," but you do not mention how Ansible compares to Puppet, Chef, or Salt for on-boarding of Mac users.
one thing that tripped me up for a bit was the term introduction... to me introduction is not a heading of a chapter on installation, basics, etc...; it seemed more like it would house an explanation for the motivation for the project.
I think it would be more aptly titled "First Steps", "Basics", or something along those lines.
Great project though! I like the methodology of it compared to chef.
Ansible is my choice too, but here's the things that trip me up or could be improved. Apologies in advance for not raising them in the proper manner. (mailing list, issue list)
The inventory docs is a single page. Specifying a directory or a host file as an inventory seem to work but it's quite unclear how the two differ in behaviour. One problem with specifying a directory is that everything in the directory is interpreted; I had a README.md in there which resulted in baffling errors.
Dynamic inventory is also way under documented; a brief intro then two giant examples (ec2 and cobbler). It needs a tiny 'hello world'-ish example. My first attempt produced output in 'ini' format'. It was almost by accident that I tried json and happened across something that worked.
But error reporting is the worst. So often ansible will produce error messages like 'please specify an inventory' when you did specify an inventory. The problem is that you misspelled the inventory file or it couldn't be read. No mention of the file's name or whether even any attempt to read it is made.
(tangentially - This latter problem seems to be endemic to Python and Java and perhaps most languages that have exceptions)
This speaks to me as I coincidentally decided to start automating my provisioning to standardize to the same vagrant environment.
I'm not a sysadmin or servers guy, but these days I happen to manage most of the devops and so getting servers and machines set up is my responsibility.
Ansible docs are not great for getting started, and I pretty much just learned by skimming through some Playbook, and figuring it out their — often intuitive — purpose.
After a few hours I got to a close to getting my machines set up with most of my Django stack: Nginx, Gunicorn, Redis, supervisor, celery, etc. The only thing that I couldn't properly set up was PostgreSQL. And here is something that most of these automation tools lack: debugging. When my postgres roles was failing, I had no clue how to even start debugging it, so after an hour or so, I just stepped back and went in to set it up manually.
I feel some of these tools need to be more friendly with devops that don't have quite the same knowledge and experience than the target audience of sysadmins and ops people. That said, I learned a ton starting to use Ansible and automated a big chunk of the process.
I've looked into devops tools for administering my personal collection of a dozen machines or so, and don't understand what they offer. Every one I've tried, I've come to the conclusion that it is primarily designed for dealing with large numbers of homogeneous machines, and only serve to add overhead (new syntax, a level of file name indirection) when dealing with individualized ones.
What I've ended up doing instead is keeping a simple tree of config files, which get run through a templating engine and then rsynced to the target host. For simple things there is zero overhead - etc/hosts is just the plain file that goes to all machines. But this system allows me to have eg one smartd.conf which contains all my disks, which is then split between machines.
A few things currently fall through the cracks (users,groups), but overall rebuilding a machine is a relatively painless process. I feel like there must be a better more standardized way though (which is why I keep looking at these tools). Any comments/experiences/etc ?
24 comments
[ 5.5 ms ] story [ 47.3 ms ] threadAs it might be useful for some I have next aliases set:
On of the (slightly) annoying things is after bringing up a new node you have to "bootstrap" the node so you can issue ansible commands on it, I have a simple script [1] for doing so but I'd love to hear how others do it![1]: https://gist.github.com/larrybolt/85b59f47615be9fcb643
This is why Ansible modules (and most other config tools) instead describe the desired end state, and perform the commands needed to get you to those states.
Basically all config tools arose out of a need for solutions to replace parallel scripting with something more robust that accounted for potential variance between systems, and also to make things more push-button.
The hosts/inventory in ansible as a source of truth for your infrastructure is also a nice feature. You could use a selection from your ansible inventory to open a cssh with 'top' running on all your webservers, for example.
This way, launching your test VM is just a `vagrant up` and running ansible against it (repeatedly, if needed) is `vagrant provision`. No need to pass inventory/key parameters at the shell this way - Vagrant calls Ansible with the right settings.
One thing that's easy to miss is to make Ansible use sudo when running against the Vagrant VM, since the vagrant user by convention has passwordless sudo rights (ansible.sudo = true in the Vagrantfile).
http://docs.vagrantup.com/v2/provisioning/ansible.html is a helpful reference doc.
You can put any given set of playbooks and configs into git and share with your team so that everyone is using ansible the same way. And that repo won't even carry any auth info.
If using metal installs, you can also set this up in your preseed or kickstart.
However, you should never have to bootstrap a machine with another script to install things like python-apt, as you can just use the "raw" command to do this for anything that is not part of the box, see http://docs.ansible.com/raw_module.html - so it's still self-bootstrapping.
disclaimer: creator of project.
And thank you for the work you put into ansible sir!
Ansible's out of the box experience allows you to start with some simple but useful constructs then allow you to graduate to more and more complex usage.
I started with some of the one-liners here, then setting up inventory files, then roles, some simple tasks and then (what really sold me) was templates.
Even just having those configuration files in an SCM is a huge win, but I honestly squealed with joy when I was able to write a nginx configuration file that dynamically included backend servers given a particular role. And that provisioning a new backend server took a couple minutes and was automatically added to the load-balancing proxy! (not to mention was instantly sending metrics to a graphite/statsd backend because collectd was installed automatically too as part of a common role and configured to point to that log collecting server!)
Magic!
Just compare http://www.ansible.com/ with the excellent http://www.vagrantup.com/ when it comes to downloading and running your first commands. After clicking around for a good while on the Ansbile site you finally find "To install on a Mac, make sure you have Homebrew, then run... ", yeah, no thanks. Oh, there's a "pip" command, if I happen to have that installed (hint: nobody on Mac has by default), but at least there's some more magic commands I can run to install that first. After running the "Looks Totally Safe To Me! (TM)" command "sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install ansible" it blows up on Mavericks with some error about something called paramiko and headers not being available. The attrition rate for newbies on Mac must be phenomenal, because you have to be really dedicated to give it another shot after this. What I'm getting at is, Python software distribution is an abysmal experience, always has been, and you should not subject your users to it. They're not Python developers, they're largely not even devs at all, they don't care about your pip's and brews.
Compare this with Vagrant: Double click this here DMG, run "vagrant init", "vagrant up" and done, in nice _big_ friendly letters on the web site.
Disclaimer: I use and love Ansible (on Linux), and want to see it succeed. Telling my Mac using ops team to check it out soured everyone's first impression right from the start. Rant over.
Finding the Ansible docs (http://docs.ansible.com/) was not as easy as it could be, however, the install instructions were easily accessible from there, exactly where I'd expect them: Introduction > Installation > Latest Releases via Homebrew.
Second, yeah, pip sucks, I try to avoid it at all costs. However, homebrew (http://brew.sh/) is that one thing that all OSX power users should install as soon as they touch their workstation. 90% of the software - CLI or GUI - on my MacBook is installed and managed by homebrew. I cannot recommend it highly enough. In fact, I didn't even install Ansible through their instructions. I read about it somewhere else, and thought, "Hmmm, I wonder if this is in homebrew? `brew search ansible` Oh! Sweet! `brew install ansible`" Worked perfectly the first time, and I haven't fiddled with it since.
I'm sorry you had install problems and had trouble reading our install chapter: http://docs.ansible.com/intro_installation.html ... I will say the docs are open source, so if you want to add a note about how to get something easier compiled on OS X or something, click the GitHub link in the upper right and you're off and running :)
We get a lot of strong positives about how awesome our docs are, so I'm really interested in what you think we could do to improve them -- but they were absolutely not modelled on Chef and Puppets. Chef has about 1000 pages of docs, Ansible covers the majority of everything in a dozen pages. Most folks get up and running (with working playbooks running real applications) in 30 minutes or an hour.
My philosophy on docs are mainly to address things in the order folks will want to learn things, but also make it easy to hop over sections people will not. Show examples, don't make a reference manual. Though there's a need for reference content, which is where stuff like galaxy.ansible.com content and github.com/ansible/ansible-examples steps in, so you can see it put together. More tutorials I think would still be useful in the main docs. But yeah, a discussion on what you like to see differently would be interesting.
While I do understand you have to do something to install the program, what you have to do to configure it once installed is pretty short, and the install itself should still only take a few minutes. I'll give you the fact that Macs are not Linux, but they are getting there :)
I'm interested in your Mavericks error and you should definitely stop by the list about that one, or just consider the homebrew install.
Another great thing to do since you are already are on a Mac is just start doing your development on Linux in a VM :) Not that we require it, we don't... but it's great when you've got a real package manager stock!
Another easy thing you can do is just checkout the git source and do "source ./hacking/env-setup" and then you are running straight out of source. This is what I do on OS X most of the time, and we have a large number of users that follow the development branch full time.
You should absolutely think of your software, even though it's free, as having a conversion funnel to happy users. You are competing both against established players like Chef and Puppet, as well as new up and comers like Salt. Once a potential user reaches ansible.com, at every single step of the on-boarding that follows you will have some % of attrition to your competitors and apathy. The more steps, the larger the attrition, the smaller your userbase is.
Think of it this way; if you add a new feature, you may gain some small percentage of users. But if you reduce the on-boarding attrition rate from 30% to, 1-2%, you will gain a simply massive amount of users over time. Do you know or track your conversion rate?
The best docs is no docs. When curious Mac user visits http://www.vagrantup.com/ they see the download button, they click the Mac installer, and they're done. At no point during the on-boarding does installing the software require even seeing the docs. Mac users and even devs expect DMGs these days. Everything from 'git', to 'vagrant', come in DMGs. On most sites (like git), you visit the front page and it autodetects you are on Mac and gives you the DMG, which every Mac user will immediately know how to use. Git comes installed by default in Macs these days. Most people on Mac never need to install git anymore. The DMG file for git on Mac has 11,000+ weekly downloads on git-scm.com (that's in addition to the brews and ports of the world, which I would imagine have less users). How would you like an extra 11K downloads a week? ;)
The tide has definitely turned in favor of Macs over the last 3 years in software shops, I see more and more developers using Macs exclusively for development. In my company, although all software gets deployed to Linux (on AWS), of the several hundred devs working on the software I would be surprised if more than a handful (like me) still use Linux on their development machines (deploying to Vagrant locally and the like). While many of the Mac converts are coming from Linux, most new devs are just given a MacBook Pro from day one. In this new reality, I fear that you are doing yourself a big disservice by trying to shoehorn the Linux software distribution models on the new Mac users.
People used to tell it is possible to bootstrap Ansible via Vagrant on Windows:
1. define a linux box as ansible control vm, provision it using vagrant's shell provider 2. mount the playbook and run it
Well, I have spent hours on this. There were errors with Python in Linux (I forgot which, but I could solve this via searches in news groups), mounting rights, but most importantly I couldn't get the ssh part working in a multi-machine setup. I still don't understand how this could happen without Ansible halting, but Ansible did apply all the remote playbooks on the control vm itself. Even whith explicitly different ip's specified.
In the end I did gave up, having wasted so much time on a productivity tool. I am ready to accept this is due to me being stupid, but other tools don't make me look as such which is better for my self esteem.
I'll have a new try soon as Vagrant and Ansible keeps maturing and maybe I can find a tutorial this time which works with the current state of Ansible.
I think that it's curious how you keep referring to Vagrant as an example of "All That is Good And Proper(tm)," but you do not mention how Ansible compares to Puppet, Chef, or Salt for on-boarding of Mac users.
I think it would be more aptly titled "First Steps", "Basics", or something along those lines.
Great project though! I like the methodology of it compared to chef.
The inventory docs is a single page. Specifying a directory or a host file as an inventory seem to work but it's quite unclear how the two differ in behaviour. One problem with specifying a directory is that everything in the directory is interpreted; I had a README.md in there which resulted in baffling errors.
Dynamic inventory is also way under documented; a brief intro then two giant examples (ec2 and cobbler). It needs a tiny 'hello world'-ish example. My first attempt produced output in 'ini' format'. It was almost by accident that I tried json and happened across something that worked.
But error reporting is the worst. So often ansible will produce error messages like 'please specify an inventory' when you did specify an inventory. The problem is that you misspelled the inventory file or it couldn't be read. No mention of the file's name or whether even any attempt to read it is made.
(tangentially - This latter problem seems to be endemic to Python and Java and perhaps most languages that have exceptions)
I'm not a sysadmin or servers guy, but these days I happen to manage most of the devops and so getting servers and machines set up is my responsibility.
Ansible docs are not great for getting started, and I pretty much just learned by skimming through some Playbook, and figuring it out their — often intuitive — purpose.
After a few hours I got to a close to getting my machines set up with most of my Django stack: Nginx, Gunicorn, Redis, supervisor, celery, etc. The only thing that I couldn't properly set up was PostgreSQL. And here is something that most of these automation tools lack: debugging. When my postgres roles was failing, I had no clue how to even start debugging it, so after an hour or so, I just stepped back and went in to set it up manually.
I feel some of these tools need to be more friendly with devops that don't have quite the same knowledge and experience than the target audience of sysadmins and ops people. That said, I learned a ton starting to use Ansible and automated a big chunk of the process.
What I've ended up doing instead is keeping a simple tree of config files, which get run through a templating engine and then rsynced to the target host. For simple things there is zero overhead - etc/hosts is just the plain file that goes to all machines. But this system allows me to have eg one smartd.conf which contains all my disks, which is then split between machines.
A few things currently fall through the cracks (users,groups), but overall rebuilding a machine is a relatively painless process. I feel like there must be a better more standardized way though (which is why I keep looking at these tools). Any comments/experiences/etc ?