we too use this. we have a waterfall build tool called Merx (custom in house, hopefully soon OS) that makes nightly builds from out dev and stable hg repositories.
Thhe builds themselves go into a package repository and on a machine we just run hg pull packages/<tool>; hg merx <machine name>; hg update;
(hg merx pulls customised machine configuration files from the repository or defaults if non-existant).
It takes a bit of initial setup but then works well for ongoing deployment (we add 10-15 commodity servers to our pool each week = easy to do now - dd the debian image and run those three commands. sorted).
We use puppet for configuration that varies between systems, but otherwise bundle our software as debian packages so just need to run 'apt-get dist-upgrade' per machine. Separate private repositories for testing/staging/production sets of the packages.
We use puppet for managing packages and configuration on our systems. At some point I'd like to take a look at Chef, too -- having Ruby as the configuration language seems like it'd be nicer than puppet's config file format. It took some work to get set up, but it's so nice being able to keep track of what's on each machine in version controlled source files.
For actually deploying our code, we use Capistrano, which we also use to run one-off tasks across machines. This pulls from our code repsitory, and each of our dev/staging/production environments know which version control branch to pull and deploy in that environment.
What do you use to produce dpkgs? In addition to the cumbersome standard tools, I'm only aware of jpkg (http://code.google.com/p/jpkg-library/) for Java.
I've found the dpkg method to be very effective. You can sign the packages, mirror the repositories, and set up servers to automatically upgrade some/all of the packages via cron jobs.
Packages can be automatically generated by your continual integration system, which also has the nice effect of creating a fully-trusted and centralized deployment path, making it difficult to sneak in uncommitted changes and non-managed files, etc.
The packages themselves support complex dependencies, start/stop scripts, etc, and dpkg/apt-get themselves have been ported to non-Linux platforms.
Personally, I'd still like a more comprehensive centrally managed solution based around the packaging model (ie, each server runs a management agent that can install/remove/update packages, etc).
I am just finishing up something akin to what you described in your last line (for my employer). It's just a simple daemon written in Python that is told what to do via XML/HTTP. Implementing such a daemon was made much easier by using the python-apt module.
I use capistrano and in some cases i use debian packages. As the debian tools don't work on OS X. I've made a debian package tool called apt_fu that can run on any unix system. It's at github:
We use an absolute minimum in terms of packaging (#) - after all, we don't need to don't need to go out of our way to make it easy to re-build. Packages are mostly C/C++ and python apps, so fortunately none of the extra issues with Java packaging.
I use fabric too, it's easy because most of the developments done here are in Python and writing a "deployment script" in the same language makes sense. The only problem I get with this is the "socket closed" error from time to time.
I use Git in with a detached working copy a la http://toroid.org/ams/git-website-howto but with a little more flair for schema updates and things like that.
We don't use source control on our production machines (which I think is crap), so I use an rsync script from my dev machine. Many people just scp the files over every time they make a change. (There was once a tendency by a few folks to directly edit files on production, without ever checking these changes in to SVN; thank god this has decreased over time.) Work is also too cheap to buy SQL migration tools for everyone who needs them, so database changes are applied by hand. Not awesome.
I think chef has a lot of potential. However, I found the learning curve rather high. If you end up using it, you'll definitely want to use a VM with snapshots.
You should spend time understanding all of the given capistrano recipes before you should start building any of your own since many things are already done for you.
I wasn't able to do this without staring at the capistrano code for a significant amount of time, but it was worth the effort IMHO.
Let me add here that darcs' inherent cherry picking abilities are a wonderful match for web development. Being able to fast track changes to the live site with no hassle is heavenly.
27 comments
[ 3.9 ms ] story [ 70.4 ms ] threadThhe builds themselves go into a package repository and on a machine we just run hg pull packages/<tool>; hg merx <machine name>; hg update;
(hg merx pulls customised machine configuration files from the repository or defaults if non-existant).
It takes a bit of initial setup but then works well for ongoing deployment (we add 10-15 commodity servers to our pool each week = easy to do now - dd the debian image and run those three commands. sorted).
http://weblog.jamisbuck.org/2007/7/23/capistrano-multistage
For actually deploying our code, we use Capistrano, which we also use to run one-off tasks across machines. This pulls from our code repsitory, and each of our dev/staging/production environments know which version control branch to pull and deploy in that environment.
I've found the dpkg method to be very effective. You can sign the packages, mirror the repositories, and set up servers to automatically upgrade some/all of the packages via cron jobs.
Packages can be automatically generated by your continual integration system, which also has the nice effect of creating a fully-trusted and centralized deployment path, making it difficult to sneak in uncommitted changes and non-managed files, etc.
The packages themselves support complex dependencies, start/stop scripts, etc, and dpkg/apt-get themselves have been ported to non-Linux platforms.
Personally, I'd still like a more comprehensive centrally managed solution based around the packaging model (ie, each server runs a management agent that can install/remove/update packages, etc).
http://github.com/sunkencity/apt_fu/tree/master
it's really a quick hack, but it get's the job done.
Oh, and dependencies are great.
(#) Link has gone now, but available at http://web.archive.org/web/20060925094756/http://www.kclee.d...
So far it meets my needs well. The only issue I had was getting my SSH key working, but I was eventually able to iron that out.
In general, I recommend against "git pull" unless you are intentionally trying to merge something. (Fast-forwards are a special case of merging.)
So my deployment looks like:
I think chef has a lot of potential. However, I found the learning curve rather high. If you end up using it, you'll definitely want to use a VM with snapshots.
Its pretty close to this: http://www.netfort.gr.jp/~dancer/software/dsh.html.en
You should spend time understanding all of the given capistrano recipes before you should start building any of your own since many things are already done for you.
I wasn't able to do this without staring at the capistrano code for a significant amount of time, but it was worth the effort IMHO.
Also a few custom tweaks to restart the Puppet daemons when they die, which they do frequently.