...and then it crashed

Programming the web with Python, Django and Javascript.

How to turn nginx into a caching, authenticated Twitter API proxy

Very soon, the old Twitter 1.0 API will be turned off, making a switch to the 1.1 API essential. Unfortunately, the new API has a couple of restrictions that can make the transition very difficult.

  • Mandatory authentication HTTP headers – Using JSONP is now impossible.
  • Restrictive crossdomain.xml – Using CORS is now impossible.

The result of these changes is that it is now impossible to access the Twitter API directly from the browser.

So, just use a proxy, right?

A simple solution is to write your own proxy server, which can then run on your own domain. The minimum features for a useful Twitter API proxy are:

  • Adds the required authentication HTTP headers to your request.
  • Caches the results to avoid exceeding API rate limits.

Writing a Python/Ruby/PHP script to handle this is easy, but it’s a waste of valuable server resources. Far better to let nginx, the best caching reverse proxy server in the world, do the hard work instead.

Step 1 – Create a Twitter application

Creating a Twitter application allows you to authenticate with the API. Just visit https://dev.twitter.com/apps and register your application with Twitter.

Once your new app is created, head over to it’s detail page and make a note of the consumer key and consumer secret. You’ll need these for the next step.

Step 2 – Obtain a bearer token for the application

The easiest way to authenticate with the Twitter API is to obtain a bearer token for your proxy server, which is a simple code that can be sent as a HTTP header with every request.

To obtain your bearer token, run the following shell commands, substituting your own consumer key and consumer secret.

1
2
3

$ export CONSUMER_KEY=XXXXXXXXXXXXXXXXXXXXX
$ export CONSUMER_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ curl -H "Authorization: Basic `echo -ne "$CONSUMER_KEY:$CONSUMER_SECRET" | base64`" -d "grant_type=client_credentials" https://api.twitter.com/oauth2/token

After a few seconds, your terminal will print out a JSON string containing your bearer token. It will look something like this:

1

{"token_type":"bearer","access_token":"AAAAAAAAAAAAAAAAAAAAAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}

Make a note of the access_token field. You’ll need this for the next step.

Step 3 – Update your nginx configuration file

Simply place the following settings in your nginx configuration, adjusting paths as necessary. In particular, make sure that proxy_cache_path, server_name and root are all correct. Most important of all, replace the INSERT_YOUR_BEARER_TOKEN placeholder with the bearer token you obtained in step 2.

(nginx.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

# This defines a 10 megabyte cache for the proxy service, and needs to live
# outside of the virtual host configuration. Adjust the path according to
# your environment.
proxy_cache_path  /var/cache/nginx/twitter_api_proxy levels=1:2 keys_zone=twitter_api_proxy:10m;

# The virtual host configuration.
server {

  # If your want to secure your proxy with SSL, replace with the appropriate SSL configuration.
  listen 80;

  # Replace this with the name of the domain you wish to run your proxy on.
  server_name api.twitter.yourdomain.com;

  # Replace this with your own document root.
  root /var/www;

  # This setting attempts to use files in the document root before
  # hitting the Twitter proxy. This allows you to put a permissive
  # crossdomain.xml file in your document root, and have it show up
  # in the browser.
  location / {
    try_files $uri $uri/index.html @twitter;
  }

  # The Twitter proxy code!
  location @twitter {

    # Caching settings, to avoid rate limits on the API service.
    proxy_cache twitter_api_proxy;
    proxy_cache_use_stale error updating timeout;
    proxy_cache_valid 200 302 404 5m;  # The server cache expires after 5 minutes - adjust as required.
    proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;

    # Hide Twitter's own caching headers - we're applying our own.
    proxy_hide_header X-Accel-Expires;
    proxy_hide_header Expires;
    proxy_hide_header Cache-Control;
    proxy_hide_header pragma;
    proxy_hide_header set-cookie;
    expires 5m;  # The browser cache expires after 5 minutes - adjust as required.

    # Set the correct host name to connect to the Twitter API.
    proxy_set_header Host api.twitter.com;

    # Add authentication headers - edit and add in your own bearer token.
    proxy_set_header Authorization "Bearer INSERT_YOUR_BEARER_TOKEN"

    # Actually proxy the request to Twitter API!
    proxy_pass https://api.twitter.com;
  }

}

Phew! That’s it, simply restart nginx and hit the following URL in your browser to make sure that everything is working!

https://api.twitter.yourdomain.com/1.1/search/tweets.json?q=cats

Comments