I have been trying to get a grasp on your second point, but to this day, I still haven't been able to really come to terms with it.
Is there any total noobs guide to automated deployment? Everything I have seen to this day assumes some knowledge of sysops and the whole ecosystem of that specific deployment tool (e.g. like it assumes I knew what is a 'cookbook', or where to find them, etc). It would be great if there is a guide for people like me that doesn't assume any knowledge from the reader aside from development / basic local deployment.
It's a lightweight management tool, like writing a shell script in python, but easier and targeted towards system deployment, really
you only have to get around its unintuitive calling convention (not inside the script - it's the way you use the cmd line tool) but otherwise it's a breeze
Chef/Puppet are harder (as in, you really have to fit their paradigms)
with Fabric you are writing shell scripts in python, so you can have people who doesn't know how to read complex(?) shell scripts be able to read/write/modify your deploy scripts.
It's similar to using a shell script except you're using Python and have access to all of Python's modules and fabric is written with roles and remote management in mind so it makes tasks easier. I love it and even use it to deploy my jekyl blog.
It makes it really easy to target ssh hosts and tell them to do things remotely, while also doing things locally, in one or more scripts.
Writing the functionality of Fabric in shell would take you a long time and give you something which was buggy and inflexible. Someone else wrote it in Python for you. There's no good reason to write it in shell.
This doesn't really answer the grandparent's question. "Try fabric" is good advice for someone that knows (a) what he wants to do and (b) how to do it using a bunch of shell scripts, but prefers Python instead. That's quite far from a "total noob" starting point.
This article[1] is what made chef 'click' for me. Try it out with vagrant or just solo on a vm to start. Try to forget everything you already learned and just do this example the build off that.
I think I should explain my point a little more. Chef and Puppet are a different category of tools compared to Fabric. It is comparable to the difference between a Makefile and a shell script.
Chef/Puppet are like Makefiles, it would rebuild the targets if they don't exist. If it is already in the desired state, running the script performs nothing. This is not the case with Fabric. Fabric scripts would usually always perform the required action (unless you have written checks explicitly).
Fabric is a low hanging fruit so I would suggest starting there until you really have the need to manage multiple servers and then use Chef/Puppet.
I would have a hard time recommending Chef and Puppet unless you wrap your deployed unit in a .deb or .rpm file with pre and post install scripts.
Puppet is not really great at executing a sequence of steps because it is declarative: you can always express sequence as dependencies (e.g., reload nginx when file X has changed), but for fine grained steps (e.g., create a softlink for the latest version of your code), it's less convenient.
It's also slower than a Fabric script. If I want to revert to a previous version, I just do "fab revert myapp". With puppet I would have to rerun the puppet agent/change a manifest/change the version in my rpm repository/...
We heavily use puppet to manage our servers, but we rely on Fabric for deployment tasks.
> Is there any total noobs guide to automated deployment? Everything I have seen to this day assumes some knowledge of sysops and the whole ecosystem of that specific deployment tool (e.g. like it assumes I knew what is a 'cookbook', or where to find them, etc).
You need to have some knowledge of sysops to use automated deployment tools. I guess it's why they were created in the first place, i.e. not for beginners but for those with enough knowledge/skill to automate tedious sys admin tasks. Perhaps the best solution for you is to use a PaaS like Heroku, EngineYard, Google App Engine, etc.
I am currently using Heroku for live apps, and it is indeed simple enough to use without having to do all these things, but one thing that I intend to use is to automate my VMs setup. I am currently using Vagrant to automate some system configuration such as security keys and port forwarding, but I still install all my packages manually.
I am not looking to be so proficient that I can handle a production server, but I do wish that I can automate some of my development workflow.
For the true beginner, it's tough to beat Heroku. Their documentation on how to deploy is pretty sound. It's as simple as a "git push" to deploy your application. Once you're comfortable with that, it's not too bad to slowly move stuff over to EC2 using your own tools (eg. Fabric).
That's my understanding. Basically... these tools give you more fine-grained control at the expense of configuration & maintenance burdens. Classic PAAS tradeoffs.
Try out fabric, start really, really, really small.
I'll assume you use Git (if not, replace with your workflow). A super simple fabric script might simply be a `git push`. Instead of typing "git push origin master" in order to push to github/bitbucket/wherever, you'd run "fab deploy". Not going to go into the fabric specifics, but deploy() would pretty much just call git push.
Anyway, that may have confused you even more. My point is, start with something incredibly simple. After you've added git push, you can add to your fabfile.py to ssh into the server, and run git pull. Then add in something to run south migrations. Then collectstatic. Eventually you'll build to something that you can simply run, and everything will be updated and all good.
If you're working with python, I would recommend looking at buildout first. The best part of working with plone lately has been learning bootstrap. For working with Django and buildoout see eg:
I'd say fabric is more powerful than Chef or Puppet. Fabric basically lets you do anything and Chef and Puppet are constrained to do certain things in certain ways. Not sure why that would be considered more powerful. Also I think Chef and Puppet require production servers to do things other than production things, like check for updates. I'd rather just have production servers do production things and not unintentionally update or do things when someone didn't intentionally instruct the server to do those things (as part of an automated process of course).
Fabric does let you do pretty much anything. But what it actually does is limited to certain basic tools. Tools like Chef and Puppet don't let you do just anything, but they also do a lot more. Some of the things they do would be pretty insane to do on top of Fabric.
Even at the beginner level, Travis CI is great. Granted, it requires knowledge of git.
It’s a really effortless way of automated testing, and you can add more and more specificity to it, as you develop your project. And the merge validation is invaluable.
(Assuming you can get it to work, which hasn’t always been a cake walk with weird, inexplicable failures every now and then.)
I can't speak for Chef/Puppet but I tried in the past to incorporate Fabric into my workflow but I couldn't find a fabfile that worked out of the box with my setup so I found myself having to learn the specifics of all the deployment steps - initially for the purpose of creating my own Fabric setup. Once I got through to the other side I realized that my process really isn't that complicated to begin with, and now I really appreciate knowing exactly how my website works. Just for the record I run a single blog using Django, nginx, uwsgi and supervisord on Webfaction. I use git and virtualenvwrapper to run development on local computers and staging and production environments on webfaction. Once I get into using the same setup for many sites, I believe I will appreciate solutions like fabric or chef, but for right now I can't stress enough the value of writing your own conf files, syncing and migrating your model changes, learning to use git, and stopping and restarting processes manually.
TL;DR While Fabric is a great tool to help you simplify your process, I wouldn't recommend it to beginners of Django. Instead make a few passes at going through deployment manually and establish a process so you can write your own fabric configurations later on.
> I can't speak for Chef/Puppet but I tried in the past to incorporate Fabric into my workflow but I couldn't find a fabfile that worked out of the box [...]
You should think of a fabfile as a glorified shell script (or Makefile). So, instead of writing your own shell script that uploads your assets to your remote box, restarts Apache, etc. and then adjusting your script to run on multiple boxes, you can just write a simple Fabric task or two that does the same thing (with minimal configuration). Chef/Puppet are completely different tools than Fabric. In fact, the two are complementary[1].
I totally recommend Fabric to beginners because it builds on what you should know, i.e. how to write simple shell scripts, and it makes life easier.
I've got to disagree with the chapter about settings files. Instead of maintaining multiple different settings for different environments, make your default settings file pull modifications for each environment from os.environ, e.g.:
I use a version of that -- a single system variable identifying the environment and a series of conditional blocks for each environment in the settings.py file:
ENVIRONMENT = os.environ.get('ENVIRONMENT', None)
if not ENVIRONMENT in ('development', 'staging', 'production'):
raise Exception('You need to set ENVIRONMENT variable')
if ENVIRONMENT == 'development':
# ...
elif ENVIRONMENT == 'staging':
# ...
Import a local_settings.py that isn't in the repository. Setting os environment variables and having multiple settings in the repository seems inflexible, a potential security problem, and just introduces unnecessary complexity to your application (doing different things because of a custom os environment variable). The security issue is production instructions in your main source repo, putting those instructions in a secured repo, hopefully maybe even encrypted, with limited access is probably more secure.
I have a strong preference for actual separate, situation-specific settings files. Mostly due to long, sorrowful experience debugging web server setups which were leaking environment variables across virtual hosts.
I bought that book too. I was skeptical after reading some of the author's blog but decided to take the plunge. Just twelve bucks, so whatever. After reading it I'm certain most of those reviews are faked or the work of the author's friends.
This book is just a slideshow. The chapters are tiny, and there's very little useful content, just a bunch of "do it this way because we say so." No real justification or understanding here. Author hasn't made anything substantial with Django, and his understanding of Python itself is lacking.
People have been request a free sample chapter since the beginning, and he keeps saying that he's getting one ready. You see a free chapter, and you're much less likely to want to buy the rest.
First of all, there are two authors of the book, and I'm the one you are mentioning. Audrey Roy is the other one, I'm just the one who spends more time on Twitter, HN, and blogging.
In any case, thank you for this mini-review. We can't make the book better without feedback, including negative commentary. So far most of what we've gotten back has been extremely positive. While gratifying to our egos, it's the brutally honest critical reviews like yours that have given us the best information for improving the book. In fact, we ask for honesty in reviews of our work, and are willing to take soul-crushing criticism so we can improve it.
As for the existing reviews, with the possible exception of Barry Morrison, they were not written by our friends. We didn't know any one besides Barry who wrote them, and his private feedback to us as a technical reviewer was pretty negative in places.
I find your point about not enough justification for our practices very interesting. We talked to a lot of core/senior Django developers about them, but I think you might have a good point that we need to explain the background. I thought we addressed this in the Beta release, and I would like to hear your feelings about that version.
Some more points:
The chapters seem tiny because we self-published and there is no fluff. This time I didn't have a publisher telling us, "This chapter needs 20 pages of content, add some more content!". This happens to book authors working with mainstream publishers, which is why tech books are often so thick and heavy and sit mostly unread on shelves. Instead, because we controlled the book 100% each chapter is as long as the material needs to be. Hence your feeling that this is 'light-reading', even though it has as much content and code examples as a thicker, denser book.
What do you consider substantial with Django? Should I have built Disqus, Mozilla, Pinterest, or Instagram on my own? Or is there some other level of work that I should have done? I'm curious, because unlike some people I don't hide behind a pseudonym (I'm Daniel Greenfeld and it's easy to find information about me). Unfortunately a lot of the work I've done for the past 18 months is in private github accounts. This really sucks, and I hope to fix it this year.
As for my actual Python and Django skills, I'm humble enough to acknowledge that I can always learn more. This is part of why I worked so hard on the Django CBV documentation refactor in 2012 (especially the reference docs), to expand my own knowledge of them. The day I call myself the absolute authority is the day I stop improving, and I apply this to everything I do. In fact, I used to give a talk on this exact topic. ;-)
Finally, as for the free chapter, here goes: Since the book was released in Alpha and Beta states, we want our book in a perfect state. This is why we offered the book at a discount (O'Reilly and other mainstream publishers do the same), so early adopters could try it out and provide feedback and we could get paid for our hundreds of hours of work. In return for the early adopter discount, you and many others have given us useful feedback (not enough justification for our practices), and now we can improve.
I have read the book and I am a friend,, so I guess I'm not "qualified" but I did find the book very useful, I have it on my to do to write a full review and it will be very positive, because the material is that good. It has given me some good tips and inspired me to get on with doings some stuff I had just been too lazy to get on with. I too stand by mys stuff and while I have a Pseudonym (dartdog) I can be easily found it is not something I hid behind.
Danny, it is good work, don't let one unknown, unqualified, jerk get you down!
I enjoyed two scoops. I appreciated the terseness of it.
If I had one criticism it would be the 'do it this way' style felt a little strong. I would have liked to see a 'some people dont do it this way' star on some of your coding style choices.
I'd quite like to hear about how you marketed it - I seemed to hear about the book from multiple sources all at once - you clearly did a good job there.
Any suggestions about what coding style choices you think should have alternates?
As for marketing, we hit HN, blog, twitter, and linkin. Then the word spread and we found it in other places. It was pretty awesome. We can't wait for it to get on Amazon. :-)
This is just stupid. I haven't bought the book, I haven't read it (I don't know pydanny personally), but I teach Python and Django courses. My Django courseware has several shoutouts to best practices I got from pydanny, uses crispy-forms of which he is the godfather, etc. I see one sentence of valuable review and two paragraphs of trolling.
This is great, I especially like the discussion about Managers which I haven't used before, but I have never been impressed by the technical chops of Smashing Magazine so to see that as an endorsement is actually a bit of a drawback. :/
Using pip the way they recommend is careless. It makes your deployment dependent on: 1. a working network connection and 2. All the remote repositories being up. 3. nobody having messed with the remote repos. The 3. is admittedly less of a concern, but all 3. are beyond your control and re-downloading the dependencies on each install is wasteful.
Fortunately pip has the --find-links option which, combined with having all dependencies downloaded locally makes it possible to have a fully local install.
Most people will not need to use pip --find-links on a regular basis. Hence, it is not "careless" to fail to do so.
If you are focused on "more nines" than this kind of paranoia is well justified. Network outages are not totally unknown, sometimes PyPI goes down. But these things really aren't all that regular as you seem to suggest. Usually the network is up and PyPI is up. And we haven't had an instance of 3. yet, assuming that you trust the authors of your dependencies.
Let's please not convey that it's just completely crazy to use pip in the normal, simple way. It really isn't.
45 comments
[ 1.6 ms ] story [ 107 ms ] thread* Use South before you start deploying
* Use an automated deployment tool like Fabric or more powerful ones like Chef/Puppet
* Get a better debugger and other niceties with django-command-extensions
Of course, a lot more apps can be considered must have depending on what you are working on.
Is there any total noobs guide to automated deployment? Everything I have seen to this day assumes some knowledge of sysops and the whole ecosystem of that specific deployment tool (e.g. like it assumes I knew what is a 'cookbook', or where to find them, etc). It would be great if there is a guide for people like me that doesn't assume any knowledge from the reader aside from development / basic local deployment.
It's a lightweight management tool, like writing a shell script in python, but easier and targeted towards system deployment, really
you only have to get around its unintuitive calling convention (not inside the script - it's the way you use the cmd line tool) but otherwise it's a breeze
Chef/Puppet are harder (as in, you really have to fit their paradigms)
with Fabric you are writing shell scripts in python, so you can have people who doesn't know how to read complex(?) shell scripts be able to read/write/modify your deploy scripts.
Writing the functionality of Fabric in shell would take you a long time and give you something which was buggy and inflexible. Someone else wrote it in Python for you. There's no good reason to write it in shell.
[1] http://www.opinionatedprogrammer.com/2011/06/chef-solo-tutor...
Chef/Puppet are like Makefiles, it would rebuild the targets if they don't exist. If it is already in the desired state, running the script performs nothing. This is not the case with Fabric. Fabric scripts would usually always perform the required action (unless you have written checks explicitly).
Fabric is a low hanging fruit so I would suggest starting there until you really have the need to manage multiple servers and then use Chef/Puppet.
Puppet is not really great at executing a sequence of steps because it is declarative: you can always express sequence as dependencies (e.g., reload nginx when file X has changed), but for fine grained steps (e.g., create a softlink for the latest version of your code), it's less convenient.
It's also slower than a Fabric script. If I want to revert to a previous version, I just do "fab revert myapp". With puppet I would have to rerun the puppet agent/change a manifest/change the version in my rpm repository/...
We heavily use puppet to manage our servers, but we rely on Fabric for deployment tasks.
You need to have some knowledge of sysops to use automated deployment tools. I guess it's why they were created in the first place, i.e. not for beginners but for those with enough knowledge/skill to automate tedious sys admin tasks. Perhaps the best solution for you is to use a PaaS like Heroku, EngineYard, Google App Engine, etc.
I am not looking to be so proficient that I can handle a production server, but I do wish that I can automate some of my development workflow.
I'll assume you use Git (if not, replace with your workflow). A super simple fabric script might simply be a `git push`. Instead of typing "git push origin master" in order to push to github/bitbucket/wherever, you'd run "fab deploy". Not going to go into the fabric specifics, but deploy() would pretty much just call git push.
Anyway, that may have confused you even more. My point is, start with something incredibly simple. After you've added git push, you can add to your fabfile.py to ssh into the server, and run git pull. Then add in something to run south migrations. Then collectstatic. Eventually you'll build to something that you can simply run, and everything will be updated and all good.
http://www.slideshare.net/zerok/django-buildout-en-presentat... http://www.sophomoredev.com/2012/06/how-i-use-django-virtual...
As for managing deployments and general devops, I am currently looking at ansible.
It’s a really effortless way of automated testing, and you can add more and more specificity to it, as you develop your project. And the merge validation is invaluable.
(Assuming you can get it to work, which hasn’t always been a cake walk with weird, inexplicable failures every now and then.)
TL;DR While Fabric is a great tool to help you simplify your process, I wouldn't recommend it to beginners of Django. Instead make a few passes at going through deployment manually and establish a process so you can write your own fabric configurations later on.
You should think of a fabfile as a glorified shell script (or Makefile). So, instead of writing your own shell script that uploads your assets to your remote box, restarts Apache, etc. and then adjusting your script to run on multiple boxes, you can just write a simple Fabric task or two that does the same thing (with minimal configuration). Chef/Puppet are completely different tools than Fabric. In fact, the two are complementary[1].
I totally recommend Fabric to beginners because it builds on what you should know, i.e. how to write simple shell scripts, and it makes life easier.
[1] http://www.youtube.com/watch?v=sdkAXM36C7M (DevOps for Python: Fabric + Chef)
It is a good idea nonetheless, so you can inject the variables needed from the outside
The environ trick is nice, and while it makes it harder to shoot yourself in the foot, it doesn't prevent it.
https://django.2scoops.org/
HN discussion: http://news.ycombinator.com/item?id=5074026
This book is just a slideshow. The chapters are tiny, and there's very little useful content, just a bunch of "do it this way because we say so." No real justification or understanding here. Author hasn't made anything substantial with Django, and his understanding of Python itself is lacking.
People have been request a free sample chapter since the beginning, and he keeps saying that he's getting one ready. You see a free chapter, and you're much less likely to want to buy the rest.
In any case, thank you for this mini-review. We can't make the book better without feedback, including negative commentary. So far most of what we've gotten back has been extremely positive. While gratifying to our egos, it's the brutally honest critical reviews like yours that have given us the best information for improving the book. In fact, we ask for honesty in reviews of our work, and are willing to take soul-crushing criticism so we can improve it.
As for the existing reviews, with the possible exception of Barry Morrison, they were not written by our friends. We didn't know any one besides Barry who wrote them, and his private feedback to us as a technical reviewer was pretty negative in places.
I find your point about not enough justification for our practices very interesting. We talked to a lot of core/senior Django developers about them, but I think you might have a good point that we need to explain the background. I thought we addressed this in the Beta release, and I would like to hear your feelings about that version.
Some more points:
The chapters seem tiny because we self-published and there is no fluff. This time I didn't have a publisher telling us, "This chapter needs 20 pages of content, add some more content!". This happens to book authors working with mainstream publishers, which is why tech books are often so thick and heavy and sit mostly unread on shelves. Instead, because we controlled the book 100% each chapter is as long as the material needs to be. Hence your feeling that this is 'light-reading', even though it has as much content and code examples as a thicker, denser book.
What do you consider substantial with Django? Should I have built Disqus, Mozilla, Pinterest, or Instagram on my own? Or is there some other level of work that I should have done? I'm curious, because unlike some people I don't hide behind a pseudonym (I'm Daniel Greenfeld and it's easy to find information about me). Unfortunately a lot of the work I've done for the past 18 months is in private github accounts. This really sucks, and I hope to fix it this year.
As for my actual Python and Django skills, I'm humble enough to acknowledge that I can always learn more. This is part of why I worked so hard on the Django CBV documentation refactor in 2012 (especially the reference docs), to expand my own knowledge of them. The day I call myself the absolute authority is the day I stop improving, and I apply this to everything I do. In fact, I used to give a talk on this exact topic. ;-)
Finally, as for the free chapter, here goes: Since the book was released in Alpha and Beta states, we want our book in a perfect state. This is why we offered the book at a discount (O'Reilly and other mainstream publishers do the same), so early adopters could try it out and provide feedback and we could get paid for our hundreds of hours of work. In return for the early adopter discount, you and many others have given us useful feedback (not enough justification for our practices), and now we can improve.
If I had one criticism it would be the 'do it this way' style felt a little strong. I would have liked to see a 'some people dont do it this way' star on some of your coding style choices.
I'd quite like to hear about how you marketed it - I seemed to hear about the book from multiple sources all at once - you clearly did a good job there.
As for marketing, we hit HN, blog, twitter, and linkin. Then the word spread and we found it in other places. It was pretty awesome. We can't wait for it to get on Amazon. :-)
Fortunately pip has the --find-links option which, combined with having all dependencies downloaded locally makes it possible to have a fully local install.
If you are focused on "more nines" than this kind of paranoia is well justified. Network outages are not totally unknown, sometimes PyPI goes down. But these things really aren't all that regular as you seem to suggest. Usually the network is up and PyPI is up. And we haven't had an instance of 3. yet, assuming that you trust the authors of your dependencies.
Let's please not convey that it's just completely crazy to use pip in the normal, simple way. It really isn't.