Ask HN: How to automate Python apps deployment?
I am researching how to automate the development environment setup for a team of 15+ developers that is currently manual and requires a lot of repetitive work. In addition to that I am researching how to automate testing, integration and deployment of these web apps to different environments (staging, and production).
How would you implement this huge workflow and how would you tackle it? Any success stories of automating the deployment of similar apps (Python, JS, SASS, Config files, git cloning of projects) in other companies/projects?
The current workflow is that each developer clones all the apps (30+) and sets them up locally via editing config files, then sets up Apache for some internal URLs, and then starts working on them. Finishes work, commits to remote development branch and from their everything is just manual work: review of pull-requests and merging branches...etc. Production releases are manual as well.
I am also interested in knowing how to setup/refactor/change Apps in such away that would enable further automation to be as easy as possible.
Cheers!
19 comments
[ 3.3 ms ] story [ 58.4 ms ] threadIt handles 3 situations:
- local ( not included in Jenkins, but is automated within the development tools -- eg. Visual Studio. It uses local databases and local paths)
- development
- beta
- live / production
So my entire workflow is automated now, the merging of branches and etc. is done by the developer. Testing is automated ( unit tests) ... Deploying is just a matter of minutes instead of 1,5 hours. And best of all, all related packages are updated automaticly with the correct tag ( development, beta, ..)
Another way would be using TeamCity ( instead of Jenkins)
We use Python virtual environments to make every app's dependencies in one place and not installed globally. Also, the frontend dependencies and code compilation is handled with npm, Gulp and Bower.
The thing is the developer has to issue all these commands in order to "setup" the app locally and be able to run and troubleshoot it before doing actual work, and this is universal for all apps. Same thing goes for setting up apps on staging and production other than the pull-requests and branches-merging tasks which are all manual.
And if a new developer comes in, they will have to install and configure their workstation before doing any actual manual setup which is also an overhead. This scenario would also happen if a developer comes back from a long holiday.
Any ideas on this?
For development: use your OS's packaging system for installing dependencies and a makefile for compiling things.
Really, I don't understand why programmers tend to avoid using sysadmins' tools in their work, especially now with this DevOps thing, when programmers are supposed to see some administration themselves.
Do you think this is a good way to automate the boring workflow:
* git clone project
* make virtualenv
* edit config file
* npm install
* bower install
* exectue gulp build task
* enable apache internal url for project
* restart apache
?
No, it doesn't sound good even remotely. This way you have method for setting up things that is very brittle, probably can't be run with confidence on a machine that was already set up, and this method is not suitable for production or test deployment, so you need to maintain two setup methods.
You should look at how developers of various distributions (Debian, CentOS/Red Hat, Fedora, Ubuntu) package their software. All these foo-dev packages containing static libraries and headers, and foo packages with actual libraries and tools, either having their respective dependencies (e.g. foo-dev depending on gcc, so you'll get compiler by just running `apt-get install foo-dev').
Apache is not a good choice for development tool. It's not pretty to set up a second instance that runs without root privileges. I use uWSGI for those, I just use different communication protocols for production (FastCGI or uWSGI's own thing) and development (HTTP, to avoid setting up fully blown web server).
Running `npm install' or `bower install' or `pip install' to just set up your environment is a method for a big stall in case of network outage (see this thread: https://news.ycombinator.com/item?id=10984775), unless you prepare local caches for all these repositories. You should have all the things in your local repository, either source control repository (your stuff) or packages repository (external dependencies).
There's not much to say. You depend on external services (and network connection) to write, build, and deploy locally developed things. All this fails badly when one of the services experiences problems.
> Also what do you exactly refer to by package repository and how would installing pip, bower or npm modules become different?
Dude, really? `apt-get' didn't give a hint?
And all those modules are different from one, single package repository in that you won't install Python modules using npm, or JavaScript with pip, but you can do all those with `apt-get'. And you'll get repeatable process, since you control your package repository.
How would you clone a git repository, compile the SASS into CSS, uglify the javascript, and run a verticalization gulp task, after of course installing the frontend libraries through npm and bower all in apt-get? Oh no hold on, you won't use neither npm, nor bower to download the frontend deps... oh no, not even pip. Mind ``explaining like I'm 5``? :)
For more explaination how package systems work, ask sysadmins in your company, as HN comments is not an appropriate medium for tutorial about them. If you don't have a sysadmin, well, you need to employ one anyway.
And this is exactly what I meant when I said that programmers tend to avoid using sysadmins' tools.
We work out of a mono-repo and use pants (https://pantsbuild.github.io/) for python/jvm/js packaging to deploy on mesos (https://mesos.apache.org/). Pants gives us (effectively) the same development workflow as in production and let's us treat any application (we have ~500 at this point) as just a command line executable (for things with weird lib dependencies, we do statically compiled wheels for each architecture - CentOS and OS X in our case - on our own pypi mirror). We've got a few kinks to still workout for local development of web projects (mostly related to JS/CSS) but for things that don't touch the frontend it's pretty beautiful.
2)It's popular to run your automation through CI tools like Jenkins because it has auditing and logging built in and it's easy to set up.
3) Don't run bower/npm install/update on production because if npm or github go down then your production site will break because a partial code change has taken place. You should build (and test) on a separate server and then copy the code changes over to production.
Really you're looking for a Capistrano for python apps. If you're going to build a deployment tool on your own it's worth looking at their approach. For python, fabric is a fairly standard way to automate code deployments. The good news is that you're on the right path. Having automated deployment procedures will give devs a lot more time back AND remove a whole class of errors
I've moved my current company's deployment from manual line by line ftp deployment to an automated build and deploy using Jenkins to trigger ansible playbooks. Given the choice, I'd probably just use capistrano or fabric instead of building from scratch. I'd estimate that we've saved several months of work per a year in both deployment time and in debugging production issues.
I will take a look at Capistrano, Fabric, Ansible and SaltStack, in addition to the configuration management topic.
I think your advise of copying source packages and dependencies makes total sense, I have seen it happen at other places.
My question remains, how would I automate a developer's environment setup (machine)? Assume I have the OS installed and connected to the internet already. Can I use the tools you referred to for deploying these apps to developers machines? What would you say regarding Python's virtualenvs creation? Can that be incorporated into the configuration management tools?
https://amfarrell.com/saltstack-from-scratch/
Also let me know if any of the integration tests for any of the tests fail on your setup.
I will definitely check it out soon, but since we're doing releases and are short on time, mind explaining the general headlines of this tutorial? Say if I do everything manual from development machines setups and installation of deps and packages, to staging and production releases, what will I be able to automate after applying your solution? And what steps won't be covered? Can you give me a detailed list of these items?
Development machines setups: The tutorial walks you through running your app on virtualbox machines the run locally, so that is an option. However, it doesn't cover how to set up a development environment outside of a virtual machine.
Installation of deps and packages: This it covers thoroughly.
Staging and production releases: It does not walk you through this explicitly, but once you go through the tutorial, using the tools it teaches to do this is fairly straightforward.