HN Explorer

Python Requests: HTTP for Humans (python-requests.org)

397 points by frzn ↗ HN
Requests is an ISC Licensed HTTP library, written in Python, for human beings.

Requests allow you to send HEAD, GET, POST, PUT, PATCH, and DELETE HTTP requests. You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way. It’s powered by urllib2, but it does all the hard work and crazy hacks for you.

85 comments

[ 8.1 ms ] story [ 170 ms ] thread
leonderijke
I'm using it in a project I'm working on. It's really easy to use!
d0vs
Unfortunately, no Python 3 support
kenneth_reitz
I can assure you that it's directly around the corner :)
sneak
Why is that unfortunate? Why would anyone run python > 2.7?
baq
why not?
briancurtin
Do you have a good reason why they shouldn't?

Apparently 113,798 people found a reason to download the 3.2.2 Windows installers (32 and 64 bit) from python.org through only the first 10 days of the month of October. http://www.python.org/webstats/usage_201110.html

(doesn't include mention of the 3.2 source tarball downloads, which are under 5,000, most likely since people get that from their distro anyway)

megaman821
This looks so nice. Is there any upcoming support for HTTP caching? I know it is not the easiest thing to do because you will probably need to make several cache backends (file, db, memcached, redis, etc.) to keep everyone happy.

Also I couldn't tell if there is keep-alive support, is there?

kenneth_reitz
Keep-alive is 100% supported in the upcoming v0.7.0 release, along with async i/o :)

Caching: I think that will be best be served by a hook/helper module, but we'll see.

samps
Cool! Is there anything I can look at to see how async I/O will work? Does all the async support depend on gevent monkey patching (as requests/async.py would suggest)?
kenneth_reitz
It's unreleased, and therefore completely undocumented at the moment. But here's the API: https://github.com/kennethreitz/requests/issues/98
st3fan
Mmm async.

Does that also mean that it will have better support for timeouts? I'm not interested in socket timeouts, more in a 'finish this in N seconds or just give up' thing.

For that reason I am using pycurl now for some stuff where I had previously used the requests library. Which has a really really horrible API :-/

simonw
The most interesting thing about caching is that you also get support for 304/Etags/Last-Modified - which is amazingly useful for any application that might hit the same URL more than once. I'd vote for that logic in requests core, with caching as a key/value interface (ship an in-memory or file-based default, but make it easy to hook in memcached/redis/etc ala Django).
dreamdu5t
What's with the "for human beings" phrase? Is that a way of saying the software is easy?
RegEx
"Requests is an ISC Licensed HTTP library, written in Python, for human beings.

Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks."

jabwork
It would appear so. Based off of testimonials and briefly looking at the API I'd say this is primarily aimed at a simple API + fewer "gotchas"

E.g., urllib/urllib2 both support urlopen(url, [data]) if `data` is supplied the request is executed as a POST, if not it is executed as a GET.

Issuing a GET w/ a querystring requires mutating the `url` and omitting `data`. This surprised me at first, and probably a number of others

JonnieCache
It generally means it has an API which lends itself to writing readable, intuitive code.
scott_s
Some APIs are appropriate for programmers to target. Others are more appropriate for compilers or higher-abstraction libraries to target.
voyvf
I love this API, and it looks like the dev version is getting even better. r.iter_content is exactly what I needed! :D
old-gregg
Absolutely my favourite recent development in the Python world. I won't be mistaken if I say that many of us have built half-arse versions of this on top of urllib(2) multiple times in the past. :-)

The support for keep-alive is particularly welcome, huge kudos to the author.

Question to users of other languages: do equivalents exist for PHP, Perl, C# or Java? Ruby folks rave about restclient, but is there anything else?

bad_user
In the FAQ it says that support for keep-alive is on the way.

I do have a half-baked implementation on urllib2 that I hate. This is really cool.

kenneth_reitz
Keep-alive has first-class support in the soon-to-be-released version!
Kliment
Is it just me or does the FAQ break the back button?
davnola
For Ruby there's httparty, https://github.com/jnunemaker/httparty, curb, and a couple of others that escape me right now.
bascule
Here's the new one I'm working on:

https://github.com/tarcieri/http

httparty does a few of the same tricks, but I still find its API a bit obtrusive. At least it automatically parses JSON! (my library does this too, if an appropriate library is loaded)

In addition to a humane interface, it will soon be backed by a Ragel-generated HTTP parser:

https://github.com/tarcieri/http/tree/master/parser

bonzoesc
Did you get to re-use any of Zed Shaw's excellent parser work from Mongrel?
bascule
Zed had two parsers: a server parser as part of Mongrel and a client parser as part of RFuzz. I actually used the client parser from RFuzz in my Rev/Cool.io libraries. I talked to Zed about his plans for these, he wanted to do a unified parser that can handle both HTTP requests and responses (since the only difference between the two is the first line) but he never got around to it. Both projects are now unmaintained.

Carl is working on a comprehensive event-driven HTTP (and Websockets) framework for Clojure. I'm just borrowing his parser and re-wrapping it in Ruby. I prefer to use Carl's as he's (at least for now) actively maintaining it, as opposed to using Zed's abandonware.

samstokes
Looks very promising. I see it currently uses Net::HTTP under the hood - do you plan to keep it that way, like Httparty and Faraday, or are you going to do away with Net::HTTP altogether? (The comment saying "this is temporary" next to requiring 'net/https' is encouraging... :))
angelbob
Also Typhoeus and RestClient.

Probably others.

DrJokepu
C# / .NET had System.Net.WebRequest since 1.1. It's similar in principle to Requests.
johns
No, no it's not. (Http)WebRequest IS AWFUL. I built RestSharp because HWR is so bad. WebClient is closer to requests, but still horribly awful if you know anything about HTTP.

The new System.Net.Http.HttpClient in .NET 4.5 is closer to Requests and not entirely bad, if a little rigid.

DrJokepu
Just out of curiosity, what's so awful about WebRequest? I've been using it for years for various things and never really had a problem with it.
glassx
I'm not your parent poster but I personally dislike that you have to instantiate a separated Response object, and that you have to fetch a stream, even for tiny responses. In those Python/Ruby frameworks it's just one line. I always end up writing a wrapper just to change it.

And this one is a pet peeve, but I hate that verbose stateful OO idiom where you instantiate an object (with a factory) and then use setters to change properties, instead of setting it all in a single command. I don't know if other .NET HTTP libraries change that, though...

johns 

  - It's too low level for quick stuff
  - It handles HTTP basic auth wrong
  - It throws an exception, AN EXCEPTION, for any response that isn't 2xx
  - Async is an ugly mess
  - In love with streams
  - Horribly inconsistent across framework profiles
  - ServicePointManager.Expect100Continue
  - Inconsistent header handling and restrictions
  - Terrible cookie API
burke_holland
I think it's fine for simple scenarios (i.e. plain GET), but it breaks down in a hurry and the fact that you have to read out of the response stream is real overhead.
masklinn
> I think it's fine for simple scenarios (i.e. plain GET)

So's python's urllib2, the point of `request` is that as soon as you go even slightly beyond very basic GET and POST, it breaks down quickly in a horrible mess.

mattmanser
It's not fine for a plain GET, it's awful.

I'm still surprised every time I use it that they've not released a modern library for it in .Net.

ArbitraryLimits
The async handling was the deal breaker for me. I gritted my teeth through the other stuff but finally gave up at that point.
draegtun
For Perl, I've mentioned this elsewhere here but Mojo::UserAgent looks quite neat. This time with Mojolicious url - http://mojolicio.us/perldoc/Mojo/UserAgent
kgtm
Interesting things can happen with M::UA in a single line:

  use Mojo::UserAgent;
  print Mojo::UserAgent->new->get('http://freegeoip.net/json/8.8.8.8')->res->json->{country_name};
Very powerful.
code_duck
I always use libcurl for PHP... don't really have any complaints.
maxogden
timelines suggest that requests was inspired by https://github.com/mikeal/request from the land of node
drgath
Just about every library & language has a Request-like wrapper. It's nothing unique to Node, and certainly nothing unique to client-side JavaScript where native XHR is a nightmare.
jacoblyles
Finally, an easy way to make POST requests from Python
mp3tricord
Looks like Python got something similar to Perl's LWP https://metacpan.org/module/LWP
draegtun
I think LWP is more equivalent to Python's urllib2 (ie. both big beasts!). So this i believe is closer to what LWP::UserAgent (https://metacpan.org/module/LWP::UserAgent) does.

Also check out Mojo::UserAgent (https://metacpan.org/module/Mojo::UserAgent) for a nice(er) alternative. And don't forget that Perl now comes a nice lightweight alternative as standard: https://metacpan.org/module/HTTP::Tiny

mp3tricord
Yes gp, LWP::UserAgent specifically. Never knew about HTTP::Tiny thanks.. I'll check it out.
vidar
What is the timeline for v0.7? Desperately need the asynchronous stuff!
kenneth_reitz
As soon as I can push it out. Aiming for Friday (if not sooner)!
vidar
That is great news! I really appreciate your work, thanks!
jrydberg
It would be nice with something similar for Twisted.
efsavage
I don't mean to badmouth Requests or Python in general, it certainly looks like an improvement, but can anyone enlighten me as to why an API that wraps an HTTP request with concise methods is #1 on HN, twice? This looks more like a good answer to SO question than "news".

This looks like all the other wrapper methods you spend 5 minutes doing, once, when you start a new codebase?

mcav
It's #1 because people vote it up; people vote it up because it interests them or they'd like to see more of it. That's the pattern for every link here.

We may agree or disagree with the consensus of the masses, but the masses make the front page.

tomp
The design of this library reminds me of Apple's design... While a lot of Python is more like Google, or Microsoft - they just don't get it. (Admittedly, parts of Python are very well-designed, but I still see a lot of room for improvement...)
toyg
Maybe because doing the same wrappers over and over again, as short as they might be, is boring?
lukeschlather
The philosophy of Python is that those wrapper methods shouldn't be necessary.

I'm glad when language quirks and libraries get linked, it's good for discussion. (And more informative than a good third of the stuff that gets linked.)

drgath
I came here with the same question. Considering I come from PHP & JavaScript where the native APIs are awful, wrapping is a necessary step for sanity. Interesting that Python didn't have this until now.
lukeschlather
Python has always had it to some degree, but merging these sorts of things into the core language is what major releases are for. No language is perfect. I think the reason you see this upvoted so much is there are so many Python programmers who want to see this on the roadmap for future versions of Python (and it probably will be.)
cjus
Given that there are a great deal of service APIs in the wild this is going to lower the barrier of entry for many less experienced developers looking to prototype their ideas.
seanmcq
I note that you call gevent.monkey.patch_all() in async.

I'm still fairly new to gevent and I'm curious if others consider this good style for a library?

MostAwesomeDude
If you're using Twisted (or Tornado, ugh) then there are builtin HTTP clients which are async in all the right ways and provide similar high-level APIs for making requests. If you're using gevent, then this is what you want. If you're using eventlet, you should switch to gevent or learn Twisted.

So yeah, this is suboptimal, but not the worst thing in the world. It should really be documented though.

artlogic
Forgive me if I've missed this, but does this library have support for multipart/form-data POSTing? That's a capability in the python library that's sorely missing - especially in 2.X.
pbreit
If Python is going to include urllib or urllib2 in the standard distro, why not make it good? Http is foundational to (web) programming. I am shocked at how poor the standard httpclient implementations are in most environments.
16s
How does this compare (feature wise) to the poster library? Are they about the same? I've used poster some, but not requests... I'll have to try it now. Just curious how it compares to poster.
d0mine
`requests` used `poster` at least in the past (if I recall correctly)
kenneth_reitz
The current stable release actually has poster inside (and has for a while), but it's being taken out soon.
josephturnip
I love Requests, but it would be really nice to have some Python libraries with cert-verified SSL support. I've hacked SSL support on top of urllib2 for my projects at work (https://github.com/josephturnerjr/urllib2.VerifiedHTTPS), but it'd be really nice if libraries started prioritizing secure communications as a first-order requirement.
kenneth_reitz
This will be in requests very soon. :)
josephturnip
Then I will be porting my projects over to Requests very soon :)

It's pretty easy to implement, as it already works at the lower (socket) level with the ssl module, it just hasn't made it up the stack into urllib2 (or Requests..yet!).

nemoniac
Excellent library but they need to fix this piece of misinformation: "A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source."

This is false. A project released as GPL can also be released on another (commercial) license.

dannyobrien
Oh, what I would give for a "mechanize for humans" built on this! I love mechanize, but I dream of not having to go back to its one page documentation and the source code every time I want to do something.
objectified
Although Requests looks very nice, I've wondered why most people wouldn't simply use pycurl - http://pycurl.sourceforge.net/ ? It's stable and you get all the power that cURL has (which is a lot), and the API is fairly simple - although maybe not as pythonic as the Requests API is.

On a side note: last time I checked urllib2 (which actually drives Requests) could not do SSL requests over a proxy, which gave me the feeling that this implementation still isn't very mature.

kenneth_reitz
I'm moving off of urllib2 as we speak, to sit directly on top of httplib.

One thing at a time :)

jsilence
Trying to use the requests module yields:

  File "/usr/lib/python2.6/cookielib.py", line 38, in <module>
    from calendar import timegm
ImportError: cannot import name timegm

for me.

On two different machines, one OSX SL with Python 2.6 from MacPorts and the other Ubuntu 10.04.3 LTS with Python 2.6 via aptitude.

Can not find any solution for this via Google and its siblings.

I'm stuck :(

-jsl