HN Explorer

Ask HN: Python framework advice

10 points by sounddust ↗ HN
This has been asked on HN before, but it was a couple of years ago and the development of Python frameworks is moving rapidly.

I'm looking for advice on which Python-based frameworks we should use on a new project.

This project will have a high level of traffic (~50 million page views/day), and will be ported from another platform from scratch, but we will need to use an existing PostgreSQL database (therefore it might be preferable not to use ORMs). We'll also need memcached support, as well as templating.

We want something lightweight, flexible, that doesn't try to do too much for us or at least makes it easy to replace functionality (for example, we have our own session management and rate-limiting systems that we want to preserve).

Also important is an active community, good documentation, and proof that it can scale (that is, it's already being used by large sites).

I've looked at many frameworks, but none I've found so far seems to meet all these requirements. (web.py seems to be the most appealing to me, but doesn't seem to have any large clients; Django seems to be the most developed, but I'm concerned about the "doesn't try to do too much for you" requirement I've listed above).

Any advice would be appreciated!

16 comments

[ 3.2 ms ] story [ 48.8 ms ] thread
zeemonkee
You'll probably get a lot of replies from people saying "just use Django", but that's just because it's what they are using.

First of, going by your requirements, I'd pick SQLAlchemy for handling your database. Although SQLAlchemy includes an ORM, it's not just an ORM. It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL:

http://www.sqlalchemy.org/docs/sqlexpression.html

If you go with SQLAlchemy the next step is the choice of framework. There are a few around, but I can only provide advice on 3: Django, Pylons and Werkzeug.

Django + SQLAlchemy is frankly not worth the bother. The real benefit to Django is the apps that go with it - for example authentication, the admin site, 3rd party apps such as Pinax and so on. These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps.

Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere.

Pylons is the next in popularity - the biggest site I know of running on Pylons is Reddit. It has more flexibility in choice of components, but lacks Django's app ecosystem and documentation is sorely lacking compared to Django. That said, you can use Mako or Jinja2, both excellent template engines, and you can integrate SQLAlchemy very easily. This lack of good, up to date docs is a pain point, as is the auto-generated modules - there is a lot of generated boilerplate code and it takes a while to learn where everything goes (routing, middleware, etc).

Finally Werkzeug - not really a full framework as such, more a toolkit for building WSGI applications. This is my personal favourite, as it's well documented and designed, and lets you structure your application the way you want to. The tutorial explains how to integrate Jinja2 and SQLAlchemy.

That said, you will spend some time laying out a Werkzeug application before you get going, unlike Django and Pylons, and I wouldn't recommend it to Python newbies. There is a new microframework - Flask (http://flask.pocoo.org) built on top of Werkzeug which makes things much easier, but it's a very new development and is probably not production-ready quite yet.

alnayyir
>It has an excellent data mapper layer you can use with legacy databases, without having to drop down to raw SQL:

http://docs.djangoproject.com/en/dev/ref/django-admin/#inspe...

>These are great, but if you are building on top of a legacy DB then you won't be able to make much benefit of them, and unfortunately the Django ORM is tightly coupled with these apps.

InspectDB > migration > populate new models

My current job is converting old systems and old databases for use with Django. It's pretty easy to convert old schemas and migrate data in Django.

>Django without these apps is not IMHO that compelling, as other components - such as the template engine - are done better elsewhere.

Nobody said you have to give up those benefits.

Please don't give advice on decisions when you haven't actually dealt with what's under discussion. It doesn't sound like you've ever actually migrated anything to Django.

I'd say pylons or Django, fiddle with both, see which one catches your fancy better. Don't take the advice of others outright.

The more time I've spent with Django, the more my complaints have faded away. The more I learn about it, the more tricks and tools I find to deal with anything I encounter.

There's really no point in dismissing it just because you don't understand inspectdb and migrations.

Edit: For the actual migration process, if you're dealing with a rather large table (millions of rows) you might want to drop down to SQL for convenience/speed, but that's still relatively optional. Most of our migrations are in python + django ORM, not in SQL. (All but 2, out of ~15)

zeemonkee
I'm well aware of inspectdb and migrations. The OP however stated that he didn't want to use an ORM.

And what if you need to support composite primary keys ? The ticket on that is 5 years old with no fix in sight:

http://code.djangoproject.com/ticket/373

Frankly, the more I've used Django the more I've run up against it's limits. One of these limits is that the apps such as auth and admin are tightly coupled to the ORM and template engine.

I've given my experiences with different frameworks, the pluses and minuses of each. It's up to the original poster to decide naturally.

However, yet again any criticism of the One Holy Web Framework brings ad hominem arguments and angry rebuttals.

alnayyir
>I'm well aware of inspectdb and migrations. The OP however stated that he didn't want to use an ORM.

That part I missed, my bad.

>And what if you need to support composite primary keys ? The ticket on that is 5 years old with no fix in sight:

If it were me? I'd patch it myself. I've patched django for local use only multiple times because when I looked at the relevant ticket, they refused a fix. I've no love for the politics of the project itself.

>Frankly, the more I've used Django the more I've run up against it's limits. One of these limits is that the apps such as auth and admin are tightly coupled to the ORM and template engine.

If you're straining against it that much, I wouldn't even necessarily bother with using SQLAlchemy, it can be rather painful at times. I'd just use nodejs or http://wiki.github.com/defnull/bottle/ with an amalgamation of preferred tools.

>I've given my experiences with different frameworks, the pluses and minuses of each. It's up to the original poster to decide naturally.

That's pretty much what I said.

>However, yet again any criticism of the One Holy Web Framework brings ad hominem arguments and angry rebuttals.

Cut the martyr bullshit. It's a technical decision, not a chance for you to air your ego.

zeemonkee
Why patch Django when SQLAlchemy supports it out of the box ?

Out of interest, what do you find painful about SQLAlchemy ? It's admittedly not as easy to use "out of the box" as Django ORM, but the payback is the flexibility.

Why would I want to use node.js ? I'm quite comfortable in Python, and don't feel the need to abandon it just because I don't want to use Django. And I did recommend Werkzeug and its amalgamation of tools.

> Cut the martyr bullshit. It's a technical decision, not a chance for you to air your ego.

Air my ego ? WTF ? I've just noticed how sensitive Django fanboys are to even the slightest criticism, and will go out of their way to justify its shortcomings. It just gets annoying. It's just a framework FFS with its advantages and disadvantages.

Don't get me wrong - Django's not that bad; it just that some projects suit it better than others.

EDIT: not calling you a fanboy personally, but there is a noticeable rise of fanboyism in Django that reminds me of Rails a few years ago. Let's both try and keep the discussion both technical and friendly.

alnayyir
>I've just noticed how sensitive Django fanboys are to even the slightest criticism

The generalizations are the problem here. Stop it

>It's admittedly not as easy to use "out of the box" as Django ORM, but the payback is the flexibility.

You could say that of assembler. I'm of the opinion that if you're going to suffer SQLAlchemy, you might as well use something else entirely. Without the kind of integration Django offers, you might as well deal with the database on its own terms.

I can't even be bothered to respond to the rest of this. I'm not in the mood to feed the troll today, not on hackerne.ws

zeemonkee
Of course you go on about how painful SQLAlchemy is without any specifics, throw in a statement about "Without the kind of integration Django offers, you might as well deal with the database on its own terms" and accuse me of trolling....
stse
I'm also in the hunt for a python framework. I've had a look at several, most have already been mentioned here, like Flask and web.py. Personally I'm probably going to go with custom framework built on Werkzeug or modifying Flask.

I think Tornado Web (http://www.tornadoweb.org/) might be an interesting option for you. Should be highly scalable and is being used by FriendFeed. There were some things that I didn't like about it, but I've seemed to have forgotten what exactly.

khill
Pylons would be my tool of choice for this type of project. I prefer both SQLAlchmeny and Mako to the Django equivalents anyway but, personal preference aside, the fact that you have an existing data model and don't want something that "tries to do too much" make Django less desirable for this project.
scorpioxy
Well, I like Django too but you pretty much removed it from the equation when you said you want a framework that doesn't do too much for you. That pretty much means you want a minimal framework. So perhaps you'd want to go with web.py which if i understand correctly is what reddit uses. Or maybe you'd just want to go with something like Cherrypy and build things on top of that.
zeemonkee
Reddit uses Pylons, not web.py (it was originally written in Lisp, then moved to web.py, then Pylons).
scorpioxy
Ah ok. I missed that last part then.

But I'd still recommend web.py or something similar because it seems to me that he doesn't really need a framework, he wants to put one together from parts he prefers. Doing that with something like Django is a bit painful.

zeemonkee
It would be good to know how experienced he is with Python. I recommended Werkzeug elsewhere, which would fit that requirement quite well, but I wouldn't necessarily recommend it to someone quite new to Python.
davisp
WebOb, Routes, SQLAlchemy, Mako. My favorite framework is no framework.

The bottom line is that its a sliding scale. Django will give you a large amount of functionality for free but requires that you do things the Django way. As you move along the scale towards projects like Werkzeug you'll inevitably lose some of the free stuff but you'll be able to make a lot more choices on how things work.

You'll want to check Ian Bicking's blog for some good reading on the no-framework approach. [1] is a great place to start.

[1] http://blog.ianbicking.org/2010/03/12/a-webob-app-example/

sounddust
Quick note: The "50 million page views/day" is a typo. I meant 50 million per month!