How does your company approach code deployments? Bonus points on dealing with multiple code bases on multiple servers over multiple data centers where up-time is critical.
We have a dedicated deployment team, consisting of mostly H1B visa folks. They write some deployment and init scripts, then hand over things over to the offshore group. This ends up costing the company a lot of money and pain due to horrid process. Usually onshore QA or development comes in to fix stuff when things go wrong. Oh, we do finance-related stuff and use insecure standard passwords all over the place. WTF am I still doing here.
We've spent quite a bit of time on our deployment system at Lanyrd, based on the philosophy that deployments should be automated, fast and cheap. Being able to deploy the site quickly and without too much performance overhead encourages multiple deployments a day, which is great for supporting rapid iteration and keeping everyone productive and happy.
Currently, our deploys are managed by a combination of fabric (the Python shell automation toolkit) and puppet. Puppet ensures our servers are fully configured, fabric orchestrates everything. We also have a Jenkins CI server which can be used to deploy at the click of a button, rather than running a command-line script.
We can deploy to different environments or using different code branches using command-line arguments. For example:
$ fab e:live deploy
Deploys our 'master' branch to our live environment.
$ fab e:staging target:feature/new-login deploy
Deploys the feature/new-login git branch to our staging servers.
"target" can take anything Git can - a branch, a tag or even a commit hash.
We use south to manage our database migrations, which run as part of the deploy. Our static CSS/JS/etc assets are hosted on S3, and have their filenames rewritten to include the hash of the file contents before being uploaded - that way we can push new assets to S3 without overwriting the old ones, meaning that we don't have to worry about keeping our asset deploys perfectly synced with rolling out new HTML templates (which may depend on new CSS etc). We push new assets automatically at the start of the deploy.
We have an IRC bot that our sysadmin scripted. We pick the project we want deployed and issue a single command, it's amazing for low friction deployments.
We recently moved from dedicated to cloud servers, so we have a Chef recipe that we run using a knife command to target all web servers in parallel. The recipe looks at each of our git repos, finds ones that have been updated on the appropriate production branch, reclones them, then symlinks them into place, replacing the existing symlink to the old version. As a bonus, the deployment recipe also makes sure all permissions and symlinks are correct.
It works pretty well so far; we'll almost certainly be looking at ways to make it easier as it's currently a rather lengthy knife command.
Fabric logs into the servers and issue chef-client-command.
Chef does currently only handle the deployment part, ie pulls new code from git and refresh the app-server. I'm looking into setting up a cookbook that would be able to take a bare cloudserver and put it into production with the webapp running on it. That way I could scale faster if I would get unexpected traffic.
9 comments
[ 2.8 ms ] story [ 32.7 ms ] threadIt's just:
git push heroku master
Currently, our deploys are managed by a combination of fabric (the Python shell automation toolkit) and puppet. Puppet ensures our servers are fully configured, fabric orchestrates everything. We also have a Jenkins CI server which can be used to deploy at the click of a button, rather than running a command-line script.
We can deploy to different environments or using different code branches using command-line arguments. For example:
Deploys our 'master' branch to our live environment. Deploys the feature/new-login git branch to our staging servers."target" can take anything Git can - a branch, a tag or even a commit hash.
We use south to manage our database migrations, which run as part of the deploy. Our static CSS/JS/etc assets are hosted on S3, and have their filenames rewritten to include the hash of the file contents before being uploaded - that way we can push new assets to S3 without overwriting the old ones, meaning that we don't have to worry about keeping our asset deploys perfectly synced with rolling out new HTML templates (which may depend on new CSS etc). We push new assets automatically at the start of the deploy.
It works pretty well so far; we'll almost certainly be looking at ways to make it easier as it's currently a rather lengthy knife command.
Fabric logs into the servers and issue chef-client-command.
Chef does currently only handle the deployment part, ie pulls new code from git and refresh the app-server. I'm looking into setting up a cookbook that would be able to take a bare cloudserver and put it into production with the webapp running on it. That way I could scale faster if I would get unexpected traffic.