Communicating with RESTful APIs in Python

REST defines a way to design an API
with which you can consume its ressources using HTTP methods (GET, POST, etc) over URLs.
Interacting with such an API basically comes down to sending HTTP requests.

In this article, we’ll see which python modules are available to solve this problem, and which one you should use.
We’ll test all modules with this simple test case: we would like to create a new Github repository
using their RESTful API.

Python and HTTP: too many modules!

One known problem with Python is the abundance of modules enabling HTTP communication:

If you’re totally new to Python, this might be confusing, so let me try to clarify things a little bit.
First things first: httplib will not solve your problem.

This module defines classes which implement the client side of the
HTTP and HTTPS protocols. It is normally not used directly — the module urllib uses it to handle URLs that use HTTP and HTTPS.

So we’re down with 4 modules that could allow us to communicate with a
RESTful API (between you and me, that’s still way too much and should
be standardized).

The Github API

Reading the Github API documentation, it appears that to create a new repository,
we need to send a POST request to, with some input data (the only mandatory input beingname, a string),
and our credentials.

Let’s see how to do such a thing with urllib2, httplib2, pycurl and requests.


import urllib2, urllib
github_url = ''
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, github_url, 'user', '***')
auth = urllib2.HTTPBasicAuthHandler(password_manager) # create an authentication handler
opener = urllib2.build_opener(auth) # create an opener with the authentication handler
urllib2.install_opener(opener) # install the opener...
request = urllib2.Request(github_url, urllib.urlencode({'name':'Test repo', 'description': 'Some test repository'})) # Manual encoding required
handler = urllib2.urlopen(request)

This code is so ugly, it looks like Java. However, once we’ve
instanciated the url, the password manager, the opener, a request is
only 2 lines
of code.
With urllib2,
a request without data will be interpreted as a GET request,
and a request with data will be intepreted as a POST one. The other HTTP
methods (PUT, PATCH, DELETE, OPTIONS, HEAD) are not supported.
Plus, the documentation is horrible…

Clearly, urllib2 is not the way to go.


The httplib2 module is a comprehensive HTTP client library that handles
caching, keep-alive, compression, redirects and many kinds of authentication.

import urllib, httplib2
github_url = ''
h = httplib2.Http(".cache") # WAT?
h.add_credentials("user", "******", "")
data = urllib.urlencode({"name":"test"})
resp, content = h.request(github_url, "POST", data)
print content

Ok that seems better. The only weird line ish = httplib2.Http(".cache"). Why would I instanciate anHTTPobject?
Shouldn’t theHTTP.requestmethod be available as a function of thehttplib2module? Seems like a design flaw to me, but that’s
not a major problem anyway.

The thing is, I lied to you. The previous examples fail miserably, sending back a 401.

Michael Foord explains the mechanism of Basic Authentication in urllib2 in
his great blogpost urllib2 - The Missing Manual: HOWTO Fetch Internet Resources with Python

When authentication is required, the server sends a header (as well
as the 401 error code) requesting authentication. This specifies the
authentication scheme and a ‘realm’. The header looks like :Www-authenticate: SCHEME realm="REALM".
The client should then retry the request with the appropriate name and
password for the realm included as a header in the request

I inspected the header Github sends me back, and it appears that there is no trace ofWww-authenticatein it.

'cache-control': '',
'connection': 'keep-alive',
'content-length': '37',
'content-type': 'application/json; charset=utf-8',
'date': 'Sun, 01 Jul 2012 06:29:07 GMT',
'server': 'nginx/1.0.13',
'status': '401',
'x-ratelimit-limit': '5000',
'x-ratelimit-remaining': '4999'

That’s why, with urllib2 and httplib2, all we get is an error and a
401 status code. The credentials are not sent back after getting a 401
from the server. If we wanted to use one of these modules, we’d then
have to write some more code that would intercept the
401 and automatically send back the credentials. We certainly do not
want to do that…


One other option is pycurl, a python binding of the cURL C library, libcurl.

import pycurl, json
github_url = ""
user_pwd = "user:*****"
data = json.dumps({"name": "test_repo", "description": "Some test repo"})
c = pycurl.Curl()
c.setopt(pycurl.URL, github_url)
c.setopt(pycurl.USERPWD, user_pwd)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)

This time, the repo is created and we get a201 - Createdstatus code. Yay!
However, we can see that pycurl is very low-level:
it took us 4 lines to configure a simple request. These lines would either have to be repeated
for each request, or be bundled into a function.
We’d then have to implement someget,post… functions, which basically comes down to re-inventing the wheel.

Thus, pycurl could be an option, but kind of defeats the purpose Python high-levelness.


Finally, let’s see how to create a Github repo with requests.

import requests, json
github_url = ""
data = json.dumps({'name':'test', 'description':'some test repo'})
r =, data, auth=('user', '*****'))
print r.json

Wait, is that it? NoCreateManagerWithExtraLargeName()method call? No manual credential sending?
Well, no. requests was designed to be an HTTP high level API,
supporting all HTTP methods, SSL encryption, proxies, redirection, caching, etc.

I find it perfect for communicating with RESTful APIs, and clearly recommend it over the previous 3 modules!
Have a look at their documentation and examples if you’re still not convinced.


requests is the perfect module if you want to repeatedly interact with
a RESTful API. It supports pretty much everything you might need in that
case, and its documentation is extremely clear. (I’m looking at you, urllib2…)

最后推荐一篇写的不错的RESTful 介绍文章:


You can donate through PayPal.
My paypal id:
Paypal page:

  1. 本文目前尚无任何评论.
  1. 本文目前尚无任何 trackbacks 和 pingbacks.