HN Explorer

Ask HN: How do you install developer tools on a fresh system?

126 points by fratlas ↗ HN
I used to have a bash script I would run to install everything (libs/tools) you're average web developer would need (node, sublime, google chrome, , python libs etc etc). Unfortunately I lost it after some time. Does anyone use anything similar?

128 comments

[ 3.0 ms ] story [ 188 ms ] thread
_query
I'm using a shell script together with the nix package manager for that. The shell script just ensures that all packages are there (e.g. doing `nix-env -i fpp wget iterm2 jekyll ghc ruby nodejs composer php`). I can pin the version of all packages by configuring `NIX_PATH` to point to a specific `nixpkgs` (the package repository) commit. So that all people have exact the same versions of everything.

Package customizations like a .vimrc is also handled by nix (I recently blogged about how I do this: https://www.mpscholten.de/nixos/2016/05/26/sharing-configura...).

The shell scripts together with the package customizations (e.g. my custom vimrc) are managed by git.

chriswarbo
I also use Nix, with sets of packages managed by git. I use four "levels":

- System-wide packages, systemd services, users, cronjobs, etc. are managed by /etc/nixos/configuration.nix

- My user profile has one package installed, which depends on all of the tools I want to be generally available (emacs, firefox, etc.). By using a single meta-package like this, I can manage it using git and it makes updates/rollbacks easier

- Each project I work on maintains its own lists of dependencies (either directly in a .nix file, or converted automatically from other formats like .cabal), which are brought into scope by nix-shell

- I also have a bunch of scripts which use nix-shell shebangs, so their dependencies are fetched when invoked, and available for garbage collection once they're finished

akavel
I've recently learned you can put all the packages in one custom Nix expression in ~/.nixpkgs/config.nix, like below, then load (and reload/update/change) it with one `nix-env -i all`, or faster with: `nix-env -iA nixos.all`:

    # ~/.nixpkgs/config.nix
    {
      packageOverrides = defaultPkgs: with defaultPkgs; {
        # To install below "pseudo-package", run:
        #  $ nix-env -i all
        # or:
        #  $ nix-env -iA nixos.all
        all = with pkgs; buildEnv {
          name = "all";
          paths = [
            fpp wget iterm2 jekyll
            ghc ruby nodejs composer php
          ];
        };
      };
    }
and then keep only this one file in git. (Though still working on how to possibly also keep .inputrc, .bashrc, .profile, etc. in it.)
[deleted]
(comment deleted)
zihotki
The question asks only about *nix systems, I assume, but it worth mentioning that there is a great tool for Windows too, just in case if someone needs it - https://chocolatey.org/
j_s
Automating from-scratch setup with chocolatey: http://www.boxstarter.org/
skeoh
Looks cool but I would be uncomfortable using this over HTTP, not HTTPS. Don't really want to risk a MITM when installing software on your machine (and giving it UAC _and_ your password).
thomasreggi
I'm very interested in a way to containerize (runing them within a vm) these tools that way nothing needs to be installed, and could go easily from machine to machine, without polluting the filesystem.
ams6110
Honestly I think this is overkill. For a dev machine, I install all my tools locally that way "yum update" or equivalent keeps everything up to date easily.

Use VMs or containers for local simulation of your deployment targets.

cyptus
https://chocolatey.org/
cyptus
windows 10 comes with packet manager, so you can just execute the following statements on an fresh windows install (powershell):

http://pastebin.com/HmiqDDbi

sakopov
Interesting. This looks like NuGet on steroids.
daveslash
Yep, this is what I do for Windows systems, and it works well for me. I have a *.bat file. The first line installs chocolatety, and the subsequent lines install packages that I want. For example... http://pastebin.com/cpbdbfAN
zauberpony
Boxstarter (http://boxstarter.org/) automates setting up windows machines even more. It utilises Chocolatey to install 3rd-party software and also can install windows updates and take care for reboots etc.

Once you've set up a box-script, you may run it on a freshly installed windows, go to lunch, and when you return everything's set up.

Jedd
Use a configuration management tool (I picked https://saltstack.com/ , mostly because of the docs & community support) but there's lots to choose from - Chef, Puppet, Ansible, and so on.

There's a learning curve, and plenty of 'where did my afternoon go?' rabbit holes you can lose yourself in. But the upside is that you can have consistent, repeatable, and rapid builds, with modularity as a bonus.

Don't be afraid with any of these kinds of tools to brute force complex components if you're in a hurry - ie. ignore the pure / idiomatic way, and use the tool's primitives to dump a shell script on a remote box and then run it.

ams6110
I've found with Ansible at least, I was initially tempted to make large complicated roles for things like "application server" or "development desktop" but what ended up working much better was very granular roles such as "nginx server" and "emacs" (often just a single task such as "yum: name=nginx state=installed") that can be combined in playbooks. This makes it easier to avoid duplicating tasks in different roles, or having a lot of complex conditional cases in your roles.
dsr_
This is also the best way to do it in Chef.
snuxoll
This is the "roles and profiles pattern" in puppet, it generally makes sense.

Also, +1 for using CM to set your workstation back up, I really need to get around to writing some Puppet manifests to do that for mine

pricechild
I maintain Ansible playbooks for my own systems should the worst happen.
ChilledHands
Just to make sure I understand, your play book isn't for a VM but a host os?
pricechild
Yep. You can use 'connection: local' for example.
dvcrn
I have my dotfiles for that. Split into different categories (brew, npm, pip) together with all the config files I need. brew and brew cask (with brew-bundle [0] for Brewfile support) take care of getting all libraries and applications onto the system.

For the development itself I'm either shipping my entire config (.vimrc for example) or use systems like spacemacs, sublimious or proton that only need 1 single file to re-install the entire state of the editor.

The install script itself [1] is then symlinking everyhing into place and executes stuff like pip install -r ~/.dotfiles/pip/packages.txt.

It takes a bit of effort to keep everything up to date but I'm never worried of loosing my "machine state". If I go to a new machine all I have to do is clone my dotfiles, execute install.sh and I have everything I need.

On servers I am using saltstack [2], a tool like puppet, ansible and friends, to ensure my machines are in the exact state I want them to be. I'm usually using the serverless version and push my states over SSH to them.

[0]: https://github.com/Homebrew/homebrew-bundle

[1]: https://github.com/dvcrn/dotfiles/blob/master/install.sh

[2]: https://saltstack.com

DigitalJack
I have no strong opinion here, but I am curious to hear yours. What were the discriminators in choosing saltstack over the alternatives?

My head spins with these tools and every time I pick one I seem to eventually run into a road block that is a no-go. The most recent effort was ansible and the no-go was its strict dependency on python2.7.

dvcrn
I came from puppet and salt felt a good amount "lighter". I'm mostly a python developer and salt states written in pure yaml with jinja2 templates, or alternatively directly in raw python for more complex stuff feel like home.

The yaml makes it very easy to understand even if I come back to a state after months. It's just a list of things that should happen with a few template tags sprinkled in.

Not saying salt is better than puppet or friends. It's purely based on preference. I can't say anything to chef or ansible since I never tried them. Salt has a crazy active community and while there are things I don't like about it like the "name" of a state, it's still doing it's job just fine so I sticked with it.

guitarbill
ha, i moved from chef to ansible for much of the same reasons. at the end of the day, it doesn't matter.

(however, from my experience salt and ansible stay readable because they're yaml and not arbitrary code/DSL, whereas chef "recipes" are ruby and usually devolve into complex programs if you're not careful)

pfista
I tried boxen from github. Like others have mentioned, tools like these have a big learning curve and boxen was pretty high maintenance after setup in my experience.
falcolas
I don't do fresh installs very often, but when I do, it's generally because I want it to be a genuine fresh install. As such, I don't install a tool until I need it. I find that tools I once thought I absolutely needed are not tools or libraries that I use anymore.

I have three exceptions to this: Vim (my editor of choice), dotfiles (which I store in a git repository and put in place using stow, installed via a simple bash script), and Vagrant, so I can do development testing against a VM.

As Docker matures, I may use it in place of Vagrant, but it's not ready to fill the same role quite yet.

munificent
> I find that tools I once thought I absolutely needed are not tools or libraries that I use anymore.

Yes! A new machine is a great time to run a garbage collection cycle on my tools.

moondev
Have you tried the beta for OSX or Windows yet? I haven't needed vagrant in the slightest since it was released.
falcolas
If they can get xhyve based VMs stable when co-existing with VirtualBox, then yeah, that will go a long way towards replacing Vagrant for me.

That said, I'm not a huge fan of Docker's feature churn; having to re-install, reconfigure, and re-learn tooling that is central to my workflow every other month gets old pretty quickly.

gtirloni
That's currently blocker #1 for us when it comes to the new Docker for Mac/Win. VirtualBox works as a nice abstraction layer on top of 3 OSes (Mac/Win/Linux) which simplifies things with Vagrant. Unfortunately it seems VirtualBox has been in maintenance mode as of lately so that might have contributed to Docker's decision to go with HyperV/xhyve.
ajford
I hadn't come across stow before. I might integrate that into my dotfiles.

Right now I use a bash script I wrote to deploy my dotfiles into ~/, via symlink, including renaming existing files (after prompting the user).

I get what you mean about using that opportunity to do some spring cleaning. I find I need to do that to my Vim installation periodically too. I'll add a new plugin that looks like it will be useful, or add something for a new language or template system, then not use it enough to justify it.

seanwilson
> I don't do fresh installs very often, but when I do, it's generally because I want it to be a genuine fresh install. As such, I don't install a tool until I need it. I find that tools I once thought I absolutely needed are not tools or libraries that I use anymore.

Same. The core tools and libraries I use change more often than my development machine so when switching to a new machine it's a fun time to reassess what I need to install so I'm not carrying any baggage. I think the time automating the installation would be a net loss given how much I'd need it.

The setup required for building projects that I work on are automated using things like Vagrant so outside of that I only really need a few tools anyways.

bryanlarsen
Question: why are you doing a fresh install? If you install everything through your package manager, you don't need a fresh install to clean things up. My operating system has lived through several different machine upgrades. I just do a `cp -a` to copy the files from one hard drive to another, set up the bootloader and go.
collyw
Theoretically that may be the case, but its far from reality.
bryanlarsen
The cases where it diverges from reality generally give me more reason to use `cp -a` to install the new system, rather than less.
Symbiote
It's not the same hard drive, but the files in /var/log/installer/ show I installed Ubuntu 10.10 on 24 October 2010. It's been upgraded since, and copied to a new drive at least once.

/etc/popularity-contest.conf has the same timestamp, so I'm curious whether Ubuntu (or Debian) keep any statistics on the lifetime of a system.

tlrobinson
I find I end up with a lot of cruft and my tools of choice change over time, so I don't worry about it. A decent package manager makes this approach tolerable.

Homebrew and Homebrew Cask on OS X handle at least 90% of what I want to install.

pwnna
I've found ansible to be okay at setting up my environment. I'm able to configure everything from my zsh themes, terminal font size, window manager shortcuts, thunderbird logins, and so forth. The playbook takes about 30 minutes to run and after that I have almost everything ready.

Unfortunately I don't have a public GH repo I can point at as I don't want to expose everything I use to the internet. However the principle is the same as provisioning servers with ansible.

The only thing different I do is I use GPG keys to decrypt and untar things like thunderbird profiles rather than using Ansible vault. I restore GPG keys + SHH keys from offline, encrypted USB backups.

ajford
Have you considered using Gitlab? I really like GitHub, but can't justify the prices right now (just starting out in my career). But while Gitlab isn't as popular, or maybe quite as polished (though it's getting there fast), it does have free private repos. I've used this to store private data like this.

Check it out if you haven't already.

pwnna
I use both GL and GH regularly (I have private repo at GH as well). I'm moving towards GL slowly but my experience is that it is much slower than GH atm.
jmfayard
That's a very good reason to only install apps with a real package installer.

On OSX, this is a no-brainer with brew[1] and brew cask[2]

# On my old mac

  $ brew list
  $ brew cask list
=> then I save relevant parts for future references

  brew install npm
  brew install zsh
  brew cask install sublime-text
  brew cask install google-chrome
  brew cask install intellij-idea
[1] http://brew.sh/ [2] https://caskroom.github.io/
desdiv
>brew list

Protip: "brew list" will list all installed packages, including dependencies, which you might not want. What you probably want is "brew leaves", where it lists all installed packages that are not dependencies of another installed package.

This makes a difference in cases where a dependency is no longer needed in the latest version.

On a related note, why does the majority of package managers make the common and simple task of "list all manually installed packages" so incredibly hard?

For a fun brain twister, try to list all the manually installed packages on your system by just reading the man pages and no internet. Ubuntu is nightmare mode for this challenge.

benwaffle
On arch, looks like this will do:

pacman -Qe

stephenr
> real package installer

> brew

That right there is irony. You intended to give good advice about using reliable methods for installing software, then immediately recommend the Devils ass crack of package management tools.

devonkim
I've been using https://github.com/superlumic/superlumic for setting up my Mac machines because it supports a few more constructs that are common for Macs such as plist file modifications. Since it's Ansible-based you use YAML files to configure everything and it works well with Homebrew at least. I spend enough time working on configuration management professionally and don't want to spend any more time than I must to keep sinking time into my workstation's configuration than I have to, so more complicated DSLs / systems like Puppet, Chef, Salt are out for me despite working with those professionally.

In the future I'd like to try NixOS for managing OS X but it seems rather immature at this point for people that want stuff to Just Work primarily.

[dead] jmfayard
That's a very good reason to only install apps with a real package installer.

On OSX, this is a no-brainer with brew[1] and brew cask[2]

# On my old mac $ brew list $ brew cask list

=> save what I will need later as

``` brew install npm brew install zsh

brew cask install sublime-text brew cask install google-chrome brew cask install github brew cask install intellij-idea ```

[1] http://brew.sh/ [2] https://caskroom.github.io/

Sir_Cmpwn
I just install stuff when I run into something I need but don't have. Keeps my system slim.
greydius 

    sudo apt-get install emacs
Cthulhu_
OP asked how to install developer tools, not how to install an OS ;)
VLM
I only have two emacs installs, home and work. But I keep them in sync, kinda, using:

https://github.com/jwiegley/use-package

So my .emacs/init.el looks like

  (use-package helm-projectile
    :ensure t
    :config (helm-projectile-on)
  )
dagss
Then discover that since last time I installed emacs on a fresh computer, somebody with an insane opinion about indentation "worked on" python-mode. Struggle with it a couple of days, then try to find an old enough emacs package for my latest Ubuntu that python-mode isn't broken.

This has happened twice.

This upgrade cycle magit (my main reason for using emacs in the first place) broke all my muscle memory and added a ton of featurea I don't want or use.

I wish I could just freeze emacs development...without getting left behind when I want to install a new package... oh well...

michaelmior
I have a repository[0] that holds all my configuration and installs some language-specific tools. Otherwise I just manually install any packages I need. I may consider automating this at some point but I don't use that many tools so it hasn't been particularly onerous.

[0] https://github.com/michaelmior/dotfiles

masukomi
ditto. In addition to dot files, my repo has a `system_setup.sh` which installs everything that can be installed on the command line and sets up symlinks and and and. Every time i add a new tool to my arsenal (brew install x usually) i also add it to that file. This repo means i can be up and running on any new system in under 30 mins. Most of that time is for some manual downloads, and git checkouts I have to do too.
fratlas
Oh god I used "you're" instead of "your" and can't edit it.
orf
The horror!
fratlas
I have become what I hate :(
aug-riedinger
You only hate not being able to edit your comment on HN. So do I.
mathgeek
+1 to Bug Finding, -1 to QA Skills
treebeard901
> (node, sublime, google chrome, , python libs etc etc).

Lazy comma usage. Come on!

groaner
Could be a Perl programmer, in which case it doesn't matter at all.
JoshTriplett
If you consistently use the same Linux distribution, consider building metapackages for that distribution.

I created a set of Debian packages that depend on suites of packages I need. I download and install "josh-apt-source", which installs the source in /etc/apt/sources.list.d and the key in /etc/apt/trusted.gpg.d/ , then "apt update" and "apt install josh-core josh-dev josh-gui ...". That same source package also builds configuration packages like "josh-config-sudoers".

logicallee
>I created a set of Debian packages that depend on suites of packages I need

You created a Debian installation of yourself. Simply run "apt install josh-core josh-dev...", and Josh will be ready to start developing on the system whenever he is connected.

tracker1
I wish... I use linux, macos and windows on a daily basis. The best I can manage is to have most of my regular scripts in dropbox, and added to my path symlinked under ~/bin/OS, which actually works pretty well. Even managed to get a bash and CMD based ssh agent running script working, which was interesting (started with the windows/cmd one, and made a bash one to match the logic/intent).

Setting up conemu to autorun a script when opening a cmd, as well as configuring my bash profile. At work (more osx these days), I usually email myself the latest, and have a ~/.auth file thats 700 that I can source in to init my proxy settings (including ssh+corkscrew)... Allows me to only have to edit credentials in one location.

Shorel
Can I use that with external PPAs?

Or I have to install the PPAs before I can use the metapackage (which kind of defeats the idea of using a metapackage instead of a script)?

stephenr
While you could have third party repos installed via apt packages the problem is you can't trigger an apt update after they install and before the things that depend on them install easily.