Show HN: Enroll, a tool to reverse-engineer servers into Ansible config mgmt (enroll.sh)
This tool was born out of a situation where I had 'inherited' a bunch of servers that were not under any form of config management. Oh, the horror...
Enroll 'harvests' system information such as what packages are installed, what services are running, what files have 'differed' from their out-of-the-box defaults, and what other custom snowflake data might exist.
The harvest state data can be kept as its own sort of SBOM, but also can be converted in a mere second or two into fully-functional Ansible roles/playbooks/inventory.
It can be run remotely over SSH or locally on the machine. Debian and Redhat-like systems are supported.
There is also a 'diff' mode to detect drift over time. (Years ago I used Puppet instead of Ansible and miss the agent/server model where it would check in and re-align to the expected state, in case people were being silly and side-stepping the config management altogether). For now, diff mode doesn't 'enforce' but is just capable of notification (webhook, email, stdout) if changes occur.
Since making the tool, I've found that it's even useful for systems where you already have in Ansible, in that it can detect stuff you forgot to put into Ansible in the first place. I'm now starting to use it as a 'DR strategy' of sorts: still favoring my normal Ansible roles day-to-day (they are more bespoke and easier to read), but running enroll with '--dangerous --sops' in the background periodically as a 'dragnet' catch-all, just in case I ever need it.
Bonus: it also can use my other tool JinjaTurtle, which converts native config files into Jinja2 templates / Ansible vars. That one too was born out of frustration, converting a massive TOML file into Ansible :)
Anyway, hope it's useful to someone other than me! The website has some demos and more documentation. Have fun every(any)-one.
33 comments
[ 2.8 ms ] story [ 76.5 ms ] threadIf something can make ansible easier for me to try out like this tool while being pragmatic, I will give this a try someday thank you!
How accurate does this tool end up becoming though? Like can I just run some bunch of commands to setup a server and then use this with ansible?
Would this end up being a good use for it or would I just re-invent something similar to cloud-init on wrong abstraction. (On all fairness, one thing troubling me about cloud-init is that I would need to probably have a list of all commands that I want to run and all changes which sometimes if history command might have some issues or you do end up writing files etc. ends up being a little messy)
I haven't played that much with both cloud-init and ansible either but I am super interested to know more about enroll and others as well as I found it really cool!
Awesome stuff!
https://asciinema.org/a/765293
Enroll will automatically make use of jinjaturtle if it's on the $PATH, to generate templates.
I just saved the state of my WSL2 instance, pushed it to github. Amazingly simple.
FWIW, I was required to add the --harvest, which your quick start seems to be missing?
ie I used:
uvx enroll single-shot --harvest ./harvest --out ./ansible
Even with a tool, people will still have to understand the output, enough that they can spot situations like "this part doesn't make sense at all", "that bit isn't static", "holy crud, there's an unsecured secret", "this part suggests a dependency on this other server we didn't know was involved, and which the tool doesn't investigate".
Does it check extended filesystem labels with e.g. getfacl for SELinux support?
I've also done this more than a few times and not written a tool.
At least once I've scripted better then regex to convert a configuration file to a Jinja2 templated configuration file (from the current package's default commented config file with the latest options). And then the need is to diff: non-executable and executable guidelines, the package default config (on each platform), and our config.
Sometimes it's better not to re-specify a default config param and value, but only if the defaults are sane on every platform. Cipher lists for example.
P2V (physical to virtual) workflows don't result in auditable system policy like this.
Most of the OS and Userspace packages backed up in full system images (as with typical P2V workflows) are exploitably out of date in weeks or months.
To do immutable upgrades with rollback, Rpm-ostree distros install the RPM packages atop the latest signed immutable rootfs image, and then layer /etc on top (and mounts in /var which hosts flatpaks and /var/home). It keeps a list of packages to reinstall and it does a smart merge of /etc. Unfortunately etckeeper (which auto-git-commits /etc before and after package upgrades) doesn't yet work with rpm-ostree distros.
Ansible does not yet work with rpm-ostree distros. IIRC the primary challenge is that ansible wants to run each `dnf install` individually and that takes forever with rpm-ostree. It is or is not the same to install one long list of packages or to install multiple groups of packages in the same sequence. It should be equivalent if the package install and post-install scripts are idempotent, but is not equivalent if e.g. `useradd` is called multiply without an explicit UID in package scripts which run as root too.
I wrote a PR to get structured output (JSON) from `dnf history`, but it was for dnf4.
From https://news.ycombinator.com/item?id=43617363 :
> upgrading the layered firefox RPM without a reboot requires -A/--apply-live (which runs twice) and upgrading the firefox flatpak doesn't require a reboot, but SELinux policies don't apply to flatpaks which run unconfined FWIU.
Does it log a list of running processes and their contexts; with `ps -Z`?
There are also VM-level diff'ing utilities for forensic-level differencing.
The reason I'm asking is because I'm interested in how much confidence could be lent to this tool with regards to more old and obscure machines that have been running for years.
I run QubesOS as my workstation, so it's been really beneficial here, because Qubes is all VMs and templates. I've been 'harvesting' one Qube VM and then building another and running the manifest on the second machine. It's been working very well to align it with the first machine. Most of my testing has been visually watching the Ansible play install/configure things that I expect to see occur.
Where it falls down:
1) Systems that are so old, the packages that it detected as being installed, are no longer 'installable' on a new system (e.g the apt repositories no longer have those packages, they just did at the time of the original install)
2) Packages that were installed not via an apt repo but, say, dpkg -i (of a .deb file). Slack Desktop is a good example. Obviously the deb is not in the harvest, so it fails there.
So, there'll always be corner cases, but assuming everything that you have installed on the old/obscure machine is still something that can be installed via the usual methods, it should be okay. (If you are running a system that is so old, its packages are no longer available upstream, it's probably time to let it go! :) )
You'll want to use --exclude-path to ignore some server-specific stuff, perhaps, such as static network configuration etc. And of course, you can also comment out whatever roles are superfluous, in the playbook, before running it.
Always use --check with Ansible first just in case.
Is it expected that it does not allocate a TTY for sudo password prompts when connecting to a remote machine via SSH? How would I use it otherwise?
I'm about to make a new release and I'll set `get_pty=True` for the paramiko calls that use sudo. I'm not 100% sure if it will fix it for all use cases, but hopefully it will.
I don't know if this touches networking or not but I've always deliberately avoided configuring (most) networking on Ansible after having shot myself in the foot too many times.
I like this in theory and will give it a shot but I'm wary of getting bled out by all the sharp corner cases...
Let me break it down as to what harvest does:
1) Detects the OS and its package backend (e.g dpkg vs rpm etc)
2) Detects what packages are installed
3) For each package, it tries to detect files in /etc/ that have been modified from the default that get shipped with the package . It does't walk the whole filesystem, it looks straight in /etc for stuff here.
4) It detects running/enabled services and timers via systemd. For each of these, it looks for the unit files, any 'drop-in' files, environment variable files, etc, as well as what executable it executes, and tries to map those systemd services to the packages it's already learned about earlier (that way, those 'packages' or future Ansible roles, can also be associated with 'handlers' in Ansible, to handle restart of the services if/when the configs change)
5) Aside from known packages already learned, it optimistically tries to capture extra system configuration in /etc that is common for config management. This is stuff like crons, logrotate configs, networking settings (as you noted!), hosts files, etc.
6) It also looks for other snowflake stuff in /etc not associated with packages/services or other typical system config, and will put these into an etc_custom role
7) Likewise, it looks in /usr/local for stuff, on the assumption that this is an area that custom apps/configs might've been placed in. These go into a usr_local_custom role.
8) It captures non-system user accounts, their group memberships and their .ssh/authorized_keys
9) takes into account anything the user set with --exclude-path or --include-path . For anything extra that is included, it will put these into an 'extra_paths' role. The location could be anywhere e.g something in /opt, /srv/ whatever you want.
10) writes the state.json and captures the artifacts
So yes, you're right - it does capture stuff that many people might want to exclude if they are going to use the manifests to build other machines from that harvest (as opposed to just rebuild the same machine itself).
But you can use --exclude-path /etc/network and so on to skip the bits you don't want. You also can always comment out from the playbook.yml or delete certain roles it generates once you've run the 'enroll manifest'.
It doesn't have any knowledge of Ansible Galaxy roles/modules etc. It generates all the roles itself. I admit, many of the existing roles out there are a lot more pleasant/easier to read (especially Jeff Geerling's). I still use those myself day to day. A lot of configs out there are also good candidates for being Jinja templates with abstracted vars for separate hosts. Enroll does use my companion tool JinjaTurtle if it's installed, but JinjaTurtle only recognises certain types of files (.ini style, .json, .xml, .yaml, .toml, but not special ones like Nginx or Apache conf files which have their own special syntax).
I consider Enroll to be a good 'quick, grab it all, so I can sleep at night' method, perhaps best for DR purposes.
In terms of safety measures: it doesn't traverse symlinks, and it has an 'IgnorePolicy' that makes it ignore most binary files (except GPG binary keys used with apt) - though if you specify certain paths with --include-path and use --dangerous, it will skip its own policy. See https://git.mig5.net/mig5/enroll/src/branch/main/enroll/igno... .
It will skip files that are too large, and it also currently has a hardcoded cap of the number of files that it will harvest (4000 for /etc/ and /usr/local/etc and /usr/local/bin, and 500 per 'role'), to avoid 'b...
https://enroll.sh/docs.html#harvest
I've also published a JSONSchema of the state.json file here to help people understand its structure: https://enroll.sh/schema.html
JinjaTurtle is just chef's kiss. Definitely taking both for a spin.