Ask HN: Configuration Management for Personal Computer?
Hello HN,
Every couple of years I find myself facing the same old tired routine: migrating my stuff off some laptop or desktop to a new one, usually combined with an OS upgrade. Is there anything like the kind of luxuries we now consider normal on the server side (IaaS; Terraform; maybe Ansible) that can be used to manage your PC and that would make re-imaging it as easy as it is on the server side?
143 comments
[ 4.4 ms ] story [ 202 ms ] thread- https://github.com/lvancrayelynghe/ansible-ubuntu
It has some bells and whistles I stripped out (or wished I did).
One of the problems I haven't really solved: ongoing updates. Say I need to install a new apt package. Do I update the Ansible playbook and rerun it? I probably should. But more often, I just install it from the command line. Now my image is out of sync.
Also certain applications like Jenkins seems surprisingly hostile towards reproducible builds.
I’ve gone through a few iterations but for me it’s come down to getting things simpler so there is less to maintain.
But for personal tools, it feels a lot like using apt - but geared towards installation in the home dir (like xstow).
As far as I can tell scoop does support installs for multiple users/services too, though.
I suppose it's entirely possible to change this via draconian policies (to have the equivalent of mounting home noexec) - and conversely, for global installs - you need to abide by UAC in the form of "sudo":
https://github.com/lukesampson/scoop/wiki/Global-Installs
[0] - https://coderwall.com/p/afmnbq/homebrew-s-new-feature-brewfi...
There's some conveniences in my bashrc (add ~bin to PATH, add bash completions from ~/.bash_completions, similar for man-path etc).
Vimplug handles my (neo)vim plug-ins, so I just have a short .vimrc-file.
I've recently consolidated on asdf as a version manager for my workstation, so I no longer manually muck with ruby/python/go venv/paths:
https://asdf-vm.com/#/
I'm surprised people in this thread aren't talking about backups. I have my dotfiles stored under revision-control, like many others here. I have a couple of notes about the setup of each host I store too. But basically I install Debian packages and that's 99% of the my install.
The only stuff that I install, outside my own code, are things like the ARduino IDE, Firefox, Calibre, and they get installed beneath /opt
Despite the lack of "real" state I still take a backup every day, and that has saved me more than once.
~/ is backed up nightly and ~/.dotfiles is on github, have some other configs on there too that I don't keep in .dotfiles, not that it matters.
The remember part is just that it doesn't really do any difference in the end, I will install what I need when I need it anyway, the important part is to keep the config that I have spent hours and days making mine. :)
- https://www.qubes-os.org/
Automation needs fairly constant attention in the order of little tweaks week over week to fight the inevitable mismatch between the automation and the software networks that it interfaces with.
Do you reinstall your workstation once a week? Unlikely and probably would be an unproductive use of your time.
The way I approach it instead -
The first thing I do is list everything I need for a workstation in an orgmode text file. For why orgmode really works well for this -
http://howardism.org/Technical/Emacs/literate-devops.html
I aim to make a "computational document" that I can execute a bit at a time as I review/check each piece on installs and upgrades of the system.
To me it's really critical that there are links to the webpages and text notes about these configurations alongside the code I'm using.
I check this and the needed config files into a single git repository.
As the workstations (I use the same document for workstation,laptop, studiopc, homeserver, android) age I'm inevitably adding and upgrading bits of the git repository. I make sure that I work from the orgmode document whenever I touch any configuration.
I use Linux and Emacs (which means a big part of my software environment is in a git controlled .emacs.d) so the entire environment supports this workflow.
> Do you reinstall your workstation once a week? Unlikely and probably would be an unproductive use of your time.
It's not about how often I reinstall. It's about how many machines I work on.
I have a personal laptop, a work laptop, a work build server, a personal cloud server, and accounts on various systems. I want all of those to have mostly the same packages, with some variations (such as no GUI on the servers, and SSH on the servers).
(In my case, I ended up with a simple set of Debian metapackages that between them depend on 99% of what I need, in addition to the usual git homedir.)
- Running `defaultinstall` will install vim, git, iotop, progress, curl, and other essentials, and it runs apt-file update.
- Running `defaultinstalloptional` will pull more packages like sipcalc, cloc, woof, and other packages that I like but are not something I use on every system or do not use regularly.
- Running `defaultinstallwifi` pulls wavemon, iw, and wireless-tools; `defaultinstallprogramming` gets me interpreters and compilers that I regularly use; `defaultinstallgui` gets me GUIs that I use (audacity, gimp, wireshark, xdotool, xclip, etc.); etc.
On a new system I run whichever ones are applicable (e.g. a desktop may be GUI but not WiFi). Together with copying .bashrc, .vim/, and .vimrc, that's 90% of the setup I need for a non-GUI system. On a GUI system, I will want to set some task bar preferences. Only replacing my main system is an exception: I'll simply copy my filesystem from the old system (or, more often, physically move over the SSD), since I'll want to have the files, firefox profile, thunderbird config, etc. anyway.
Takes nearly zero time to maintain and is not dependent on anything that I don't already consider essential (Debian won't ship without bash-compatible shell or a package manager for a long time) so there is only one manual pre-setup step: copying a text file.
https://www.stavros.io/posts/provisioning-your-computer-one-...
Just write whatever change you want to make into your Ansible file, and run the provisioning command, instead of making the change directly. Done.
The only real drain is that you need to remember when you install a new app that you need to add it to your script. Beyond that, simply keeping most non-application data on a drive that didn't get reimaged with the OS (and could store things that could be linked into the system drive) handled a lot.
I finally started using Docker with ssh -X so that my host laptop just acts as a X display and the Docker instance is what I "really" work in.
The advantage is that I use docker files (dead simple templates) as my configuration - it's there in git right now. I change it when the thing I am doing gets over the hump of me bothering to add it to git - this means I don't end up with the usual cruft cluttering up my config - things only stay in my machine on reboot if I add them to the templates (which is very simple but at least forced me to think "Do I really want that again"?
Secrets are on a usb stick which means I sometimes get to do this on my wife's Mac - assuming I can chase children off it.
I put together http://www.github.com/mikadosoftware/workstation it does all this and sorta kinda walks you through set up. Someone has sent a pull request with a good fix for ssh passwords but I have put it off thinking I will just get this "last thing here working" and frankly I am ashamed enough of ignoring free contribs that I will probably add it tomorrow on the commute
I am seriously thinking of migrating it over to nix as that does look cool but really I have lots of projects - this one I just use every day.
Edit actually the two biggest advantages are
- that I have a latest and next instance - I can rebuild one while working on the other so instead of rebuilding and reconfiguring every two years its every two days - and I always have a spare instance to work on if the build goes south. I am guessing this will mean moving to a new laptop (yeah one day) will hurts less for the same reason daily releases hurt less than quarterly releases
- I am starting to ratchet up - as it's all scripts in theory my security and other bits and pieces just go up and to the right.
Getting something to play nicely with a high-dpi display and a multi-monitor setup, seemed next to impossible.
I run Nix and it's a decent experience, but a clear benefit with containerizing user space apps is sandboxing. Sure there are ways to escape docker jail, but it's definitely more sanitary to have isolated filesystems (XSS:ed Electron apps sweeping home directories for secrets etc would have to be more sophisticated).
If you're on OS X then `brew bundle dump` and `brew bundle install` will also help[1]
[0] circleci config: https://gist.githubusercontent.com/bauerd/27b24d1a3f881fe508...
[1] https://github.com/Homebrew/homebrew-bundle
Couldn't recommend it more. I started with puppet which seemed most robust at the time but I'm sure one of the others would do the same. Having all your configs in a single git repository and easy to deploy from a central point is great. And once you have that doing things like centrally managing keys (great for ssh and tinc) becomes really easy.
https://github.com/sabrehagen/desktop-environment
"Spinning up" a new computer takes me an afternoon, which I doubt I could improve on enough by using more professional tooling to make it worth the time investment.
With the above, the most difficult step in getting a new system installed is now only typing in the WIFI password, drivers and the filesystem partitions...
How is this different from say Debian's "apt" package manager?
You can specifically reproduce the system by downloading all versions of the packages you have, right?
NixOS doesn’t just manage packages, it manages the entire system holistically. You effectively have a single configuration file (or multiple) that configure every package, service, mount, ramdisk, driver, etc. If I delete everything but my home folder, my configuration file can be used to completely recreate the root and all of its configurations. Have a KVM GPU passthrough? That can be expressed. Want to override bashrc or zshrc globally? Yep. Want to setup users, LDAP, Samba, FTP? Same configuration. It goes as far as to make most files read only so that you can’t accidentally change things.
But it goes further with build reproducibility. All builds are hashed with their inputs. Every package specifies the sha256 sum of their sources. If your package builds on one machine, it shall build on another.
But it keeps going further. NixOS eschews normal shared library paths. Binaries always refer to Nix store paths in their runpath, making sure it will only ever load the version of the library it was built to load.
And there is still more. NixOS tracks generations. Any time you rebuild the system to change a config or add a new package to the root, you get a new generation that can be rolled back. The default bootloader lets you choose what generation to boot. So if you mess up your configuration, you very rarely need recovery media.
It may not seem very important, but it pretty much changes how you interact with your OS. If you want to restart from scratch, just make sure you have copies of your home folder and configs and start over. This is great for switching to LVM or deciding to use LUKS, where it may not be easy to do without reformatting.
It does take investment, but if you have multiple machines it becomes amortized. I’m running 3 machines with largely identical setups.
Sounds very similar to Docker's philosophy, where changes to the system produce incremental changes to the system image and all dependencies are strictly controlled.
What else do I need to be convinced to leave my Ubuntu system for NixOS?
https://godarch.com/
https://github.com/pauldotknopf/darch-recipes
I'm on Windows for the time being, but will give NixOS a go some time.
For any other WSL users here, it's worth keeping an eye on this issue: https://github.com/NixOS/nixpkgs/issues/30391 (NixOS being packaged up as a WSL distro).
Because some things will be missing from distro repos, packaging is, to me, quite a valuable skill. Compare, for example, the packages for autorandr for Nix and Arch [2][3]. When I started using Arch I found the Arch package fit much better within my understanding of the world. I expect most here would find the same.
Here is what is essentially the manual for Nix packaging [4]. Personally I'd say that if you're unwilling to consume that, eventually you are going to encounter a frustrating problem with Nix that you cannot solve, and potentially regret using it. Though, using a VM or potentially a container might get you around some/many of those issues. I should also mention that Nix Pills may be worth a read before beginning to use NixOS- it's what convinced me to switch, in fact.
[0] https://unix.stackexchange.com/questions/522822/different-me...
[1] https://github.com/NixOS/nixpkgs/issues/36759
[2] https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=autor...
[3] https://github.com/NixOS/nixpkgs/blob/e380ef398761206954e514...
[4] https://nixos.org/nixos/nix-pills/
I would not recommend NixOS to everyone, but I definitely would recommend it to the type of person who wants to automate their system.
All I am doing lately is install Syncthing on any new machine, add my NAS as syncing device and setup one "Linux workstation" folder that I have.
Among other things, this folder has a hundred-something-lines shell script that pretty much does the following:
- Install all of the packages that I need
- creates symlinks of my usual dotfiles, (hosted in the "linux workstation" folder) to the proper location
- Prompts me for my gocryptfs passphrase and save it on the keyring
- starts gocryptfs to unlock another folder I have on syncthing with ssh keys
- `git clone` all of my code repos
The whole thing is done in less than 10 minutes. And the best about Syncthing is the possibility of having multiple folders x devices to define. It allows me when getting a new machine to, say, sync only my "development-related + music collection" folders if it is a work machine or if I want to have my camera pictures as well, etc.
tl;dr: bash scripts lasts longer than everything else, are easy to maintain, and anybody with basic IT skills can make sense of them. Keep in mind that in an enterprise context, the build system is used and therefore maintained, constantly, and that there's typically a team doing it. At a personal/family level, it's typically you using it, and modifying it if needed, once every other year or so, with literally nobody looking at it in between. Chances are you've done and are doing enough bash that you rarely look up its syntax. When was the last time you used terraform/ansible?
Lindy Effect (https://en.wikipedia.org/wiki/Lindy_effect) says bash will outlive all other recent solution.
Full story I went through this phases:
- full crazy pxe install + Debian/Ubuntu preseed, etc... But still needed some bash scripts. It required a lot of work from one OS version to the next. There were enough changes between versions of the pxe server that I had to revisit its config every time!
- I eventually bought a laptop for one of my kid that didn't support pxe. I installed the base OS with a USB key, and realised how easy it was, and that it was significantly less painful to install half a dozen laptops this way, than fighting my pxe server. My install became USB OS install + well, my good old bash scripts!
- I got involved with ansible very early on and decided to solve both world hunger and global warming with it, but more importantly, my laptop installs. I spent many hours on this. Got it fully automated, and felt great!
- one of my kids' laptop got destroyed, buy a new one, install from USB stick, fumble to install ansible, and realise that ansible has by now changed significantly and my scripts need a lot of work. This is in the middle of the school year, while super busy at work, I just don't have time to deal with this. But, there's a great news: I still have my old bash scripts, and guess what, they still work.
- last upgrade: I went from from my old dozen of bash scripts to this: https://github.com/dorfsmay/laptop-setup-ubuntu-18.04
It is slow-ish, especially some of the manual steps, but not painful enough to make me procrastinate that I delay an upgrade by six months. More significant: I sat down with my kids and get them to upgrade their laptops (which helped a lot to fix my documentation)!
PS: I have zero local files, everything is either on a cloud drive (pCloud), or on github if I want to keep history/share it (eg: my dot files).
>Think Dockerfiles, but for bootable, immutable, stateless, graphical (or not) environments for your everyday usage.
[0] - https://godarch.com/
https://www.gnu.org/software/stow/
http://brandon.invergo.net/news/2012-05-26-using-gnu-stow-to...
It's worked really well for me. Used to use stow.