Install lpsolve for Python

Today I wanted to try out using lpsolve with the python API on my Mac (OS X 10.7) and on my linux server.

Installing it on the Mac is tricky, but the essence is described in this blog post (here I clarify a few things that stumped me for a while, leave out some of the changes mentioned there that I didn’t need to do, and update a path or two):

  • Search for and download both of:
    • lp_solve_5.5.0.15_source.tar.gz
    • lp_solve_5.5.0.15_Python_source.tar.gz

    e.g. from sourceforge (I originally got a different version of one of these, and it did something quite different).

  • The first will extract to a folder lp_solve_5.5. The second will extract to a folder with the same name, however it will only contain an extra/Python directory. Copy this extra directory into the first download’s lp_solve_5.5 folder.
  • cd into this lp_solve_5.5 directory.
  • cd lpsolve55
  • sh ccc.osx . You will get a lot of warnings, but that’s ok. This will create a bin/ directory. On my Mac it has a subdirectory osx64/, containing liblpsolve55.a and liblpsolve55.dylib.
  • sudo cp bin/osx64/liblpsolve55.a bin/osx64/liblpsolve55.dylib /usr/local/lib (this step courtesy of this blog)
  • cd ../extra/Python
  • You now need to edit, as suggested in the blog post above (here I have updated the included directories to reflect current Xcode practice):
    LPSOLVE55 = '../../lpsolve55/bin/osx64' # not ux32
        ext_modules = ...
            include_dirs = ['../..', '/Applications/', '/Applications/'],

    You may also want to update the version number ( in, which does not match the originally downloaded files (; I’m not sure which is correct.

  • python build
  • python install . This writes out …/lib/python2.7/site-packages/lpsolve55- (in my case into my virtualenv)
  • >>> from lpsolve55 import * should now work in python.

Once this was working, I also installed lpsolve on my linux (CentOS) server. I followed the same steps there, except I executed the straight ccc file rather than ccc.osx (I changed it to refer to ~/tmp instead of /tmp, since I’m using shared hosting and cannot execute from /tmp). On linux there is no need to change the file. And I did not find it necessary to copy the .a or .dylib files anywhere.

If you’re using WebFaction, you will also want to change the final install command (as described here), to:

python install --install-lib=$HOME/webapps/web_app/lib/python2.7 \
  --install-scripts=$HOME/webapps/web_app/bin \

This also worked.

Hope that helps someone out there – let me know if you have any comments.


Django and Amazon AWS Elastic Beanstalk with S3

When you deploy your first Django website to Amazon Web Service’s Elastic Beanstalk, you will face a number of problems, such as:

  1. How should I handle static files and user-uploaded media?
  2. How do I send emails with AWS?
  3. How can I refer to the same AWS application and environment from a second development computer?
  4. How can I add gcc – and specifically, bcrypt – to the AWS environment?
  5. How can I access my site’s RDS database remotely (ie. from my local computer)?

If you’re new to Elastic Beanstalk, check out their tutorial to help you get a Django 1.4 site up on AWS quite quickly. You need to sign up for an account with AWS, but otherwise it just works. It also works for Django 1.5.

This post has some extra notes which I found handy. Also, I found it unnecessary to install MySQL-python on my local machine.

1. Static files and user-uploaded media

If you follow the tutorial above through to the optional step where you set up the admin panel, you will have set up a way to handle static files. This is often the bane of using Django (for me at least). However, you will not have a way yet to handle user-uploaded media.
To test out file uploads, I added a test app which had a model with a single FileField, and registered it with the admin. With this, I could go to the admin panel of the live site and try to upload a file, and test if it worked.

Bad approach – adapt the static files approach to media

My first thought was, if static files are being loaded ok, why not copy the same approach for user-uploaded media? So I added these lines to my config file (to match the existing lines for /static):

  - namespace: aws:elasticbeanstalk:container:python:staticfiles
    option_name: /media/
    value: media/

And to

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(os.path.dirname(os.path.dirname(
                  os.path.abspath(__file__))), 'media')

And it worked! I could click on the link to the uploaded file and see it.

Except … then I tried uploading a new version of the code with git aws.push, and suddenly I couldn’t see the file any more.

So I tried a slight variant of this approach, where I only had the one staticfiles instance in the config file, and used a MEDIA_URL of '/static/media/' and similarly for MEDIA_ROOT. It worked in the same way, which is to say, it didn’t work.

I was missing an important point, explained right at the end of this blog post: “Elastic Beanstalk images are all ephemeral… This means that nothing on an instances filesystem will survive through a deployment, redeployment, or stoppage of the environment/instance.”

Good approach – S3 with django-storages

So I had to understand more about how Django stores its files. The documentation is pretty clear on this, and I was happy to learn that the MEDIA_ROOT and MEDIA_URL settings are just locations to save files used by the default file storage system. So if you use another storage system, those two settings (probably) aren’t relevant.

When you use Elastic Beanstalk you also get an S3 bucket, so the solution is to use that to store the uploaded files. You can get your bucket name from the S3 console. The bucket name is the entire string you see there, e.g. elasticbeanstalk-us-west-2-xxxxxxxxxxxx.

We will use django-storages with boto.

First, you need to install them both (and add them to your requirements file):

pip install django-storages
pip install boto
pip freeze | grep django-storages >> requirements.txt
pip freeze | grep boto >> requirements.txt

In your file, add 'storages' to your INSTALLED_APPS, and also:

    DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
    AWS_ACCESS_KEY_ID = '---key---'
    AWS_SECRET_ACCESS_KEY = '---secret---'
    AWS_STORAGE_BUCKET_NAME = '---bucket name---'

As I mentioned, you can leave out MEDIA_URL and MEDIA_ROOT now.

And that’s it! I found that this was all I needed to do to be able to upload files through the admin panel, and have them persist. You can also see the uploaded files in your S3 console.

Note this means I am using different storage systems for the static files to the user-uploaded media files. The former do not persist from one deployment to the next (but are reloaded each time), whereas the latter do.
I’m not sure if there’s a downside to this approach – I have seen Stack Overflow posts (e.g. this one) where both sets of files are put on S3.

I’ll also mention that the links to the user-uploaded files are quite long, e.g. These parameters change between deployments.

This seems to be a good way to handle user-uploaded media. In particular, the additional parameters should limit access to unauthorised users.

2. Email

I just added the usual lines to for my gmail account:

    EMAIL_USE_TLS = True  # not sure if this is needed
    EMAIL_HOST = ''
    EMAIL_PORT = 587

Then I went into Amazon’s SES (Simple Email Service) console and verified the above EMAIL_HOST_USER email address, and some test recipient email addresses. I had to log in to gmail and respond to an email from gmail that everything was ok too.  Then, in the development sandbox, my Django app could send email fine (but only to the test recipients).

3. Referring to the same AWS environment from another computer

[Edit - this is outdated with CLI v3.0; use the 'eb' command instead.] First, you need to download a copy of the Elastic Beanstalk client to your second computer (as you did for the first one).  But this time, instead of typing eb init, you need to type (on a Mac/Linux system):

cd your/Django/project/directory
git aws.config

You will then be prompted for your access id, secret, region, etc, and you should be able to use git aws.push to push to the same place as on your other computer.

4. Adding gcc and/or bcrypt

I want to use bcrypt for password hashing. Simply adding bcrypt to your requirements.txt file is not sufficient, because bcrypt needs two more things: it needs gcc, and it needs the libffi package. Your development computer has these, but the AWS server does not.  Not being at all knowledgeable about yum or yaml, it took some trial and error to work out what changes I needed to make to .ebextensions/aws.config - so to save you this trouble, here are the extra lines you need to add to the yum section:

    libffi-devel: []
    gcc: []

5. Accessing your site’s RDS database remotely

This is surprisingly easy.  You first need to tell RDS which IP addresses are allowed to connect; this is described in detail here.  The quick summary is to find the database’s “Security Groups” console in AWS, go to the “Inbound” tab, and set the rule to “MySQL”, with your local IP address (which you can get from

You can get a copy of the database dumped onto your local machine with eg.:

/Applications/MAMP/Library/bin/mysqldump -h -u ebroot -p ebdb > db.sql

The -p option will make it prompt you for your database password, which you entered when you set up the EB environment.  (I’m using MAMP, hence the need for the path to mysqldump above – you may not need this.) Do not put the port number (eg. :3306) at the end of the URL.

If you want to run your local development version of Django with the AWS RDS database, all you need to do is set the following environment variables before you do ./ runserver:

    # export RDS_DB_NAME='ebdb'
    # export RDS_USERNAME='ebroot'
    # export RDS_PASSWORD=''  # you need to remember this
    # export RDS_HOSTNAME=''
    # (HOSTNAME is the endpoint from )
    # export RDS_PORT='3306'  # also from the console

That’s assuming you are using the suggested setup in

if 'RDS_DB_NAME' in os.environ:
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': os.environ['RDS_DB_NAME'],
            'USER': os.environ['RDS_USERNAME'],
            'PASSWORD': os.environ['RDS_PASSWORD'],
            'HOST': os.environ['RDS_HOSTNAME'],
            'PORT': os.environ['RDS_PORT'],

I hope this helps someone out there get over the hurdle to using AWS.


Use Django fixtures to export and import a database

I have some Django sites, and I’ve often wondered about how to make my dev database faithfully mirror the production database (at a point in time), i.e. copy all the info from my prod database back to my dev environment.

Approach 1: Export and import the data

You would think it is simple: just export the data from one and import it into the other. If you use the same type of database (e.g. MySQL) in both environments, this should work fine.  But I use sqlite in dev and MySQL in prod. Using sqlite on the development server is quick and easy, with no need to set up MySQL (see my earlier post on some of the hoops you may need to jump through for that to work). But then there are subtle differences between the import and export formats, and you can waste a lot of time mucking around with the files trying to get it to work.

Approach 2: Fixtures

So, the second idea is: use fixtures, e.g. on the production machine:

./ dumpdata --indent 2 > all.json

and load it in again on the dev server (I use git to transfer it but there’s probably a cleaner approach):

./ loaddata all.json

Ah – another problem. The dumpdata command, with no apps supplied, dumps everything including Users, User profiles, Sites, etc. Your dev database, even if you only just set it up a second ago, will have some content already. So you will get error messages like this:

IntegrityError: Could not load myapp.UserProfile(pk=18): column user_id is not unique

You could go back and only list the apps you want to transfer the data of after the dumpdata command, but this is painful and you may not know them all if you’re using lots of third party apps (South, Django-CMS, etc).

The solution: Fixtures + some delete statements

But I can give you good news: there is a way to do it by deleting all that initial content from your dev database, and potentially making one small deletion from the above json file.

Start with a newly created database, with the tables installed (e.g. using ./ syncdb). These are the tables you need to wipe clean:

  • django_site
  • django_content_type
  • auth_permission
  • auth_user
  • south_migrationhistory if you’re using South.  Note: I am using South but did not delete the contents of this table, and it all worked. But I suspect you should delete the contents.
  • any user profiles you define (at least in Django 1.4, I’m not sure how they work in Django 1.5 yet)

In sqlite you’d do this by typing (assuming your database file is named sqlite.db):

sqlite3 sqlite.db
  delete from django_site;
  delete from django_content_type;
  delete from auth_permission;
  delete from auth_user;
  delete from south_migrationhistory;
  delete from ...;

Looks scary, but this is a brand new database anyway, right? You could easily recreate this content by starting over with ./ syncdb.

The last line is where you delete your user profiles. If you do define a user profile, the act of creating a user will also create a profile. So when the users get loaded from the fixture file, user profiles will be automatically created; when the user profiles’ turn comes to be loaded from the fixture, there can be a potential clash. If you are OK to lose all your user’s profile data on your dev machine, there is a simple solution: just delete these entries from the json file.

That’s it! I hope that helps someone out there.


Installing PIL on Mac OS

Installing PIL (the python image library) on my Mac is non-trivial. I get this error when I try to read or write a jpeg image:

IOError: encoder jpeg not available

and this error when I try to use any fonts:

ImportError: The _imagingft C module is not installed

I understand that Pillow is a much friendlier version of PIL, but even it does not help here, as its documentation simply states “Once you have installed the prerequisites” – with no further explanation of how to do that. Also, I have never used homebrew, and am not sure how it works with virtualenv, so I’d prefer not to use it.

Here are the steps I have followed, which solve both of these problems. They took a long time to discover!

  1. Uninstall any existing PIL you may have.  I’m afraid this is easier said than done. Fortunately I had installed PIL in a virtualenv, so I could just change to a new one and go from there. If you installed it using Pillow, you should be able to just pip uninstall Pillow, but I have not tried it.
  2. Install the jpeg library.  The way to do this is partly given by this stackoverflow post, though it unfortunately misses the last step, which you can find here. To summarise:
    1. Download libjpeg from
    2. Unpack it (either using the Finder or something like  tar zxvf jpegsrc.v8c.tar.gz)
    3. ./configure
    4. make
    5. sudo make install
    6. cp -r ~/Downloads/jpeg-XX/ /usr/local/jpeg
  3. Install the freetype library.  Get this from, and follow the same procedure as for jpeg above (I didn’t do step 5 and it worked fine).  Copy it into /usr/local/freetype.  This time it will work.
  4. Install any other libraries you may want (e.g. see this list).
  5. pip install Pillow

To show this works, I get this output from the final command:

    SETUP SUMMARY (Pillow 2.0.0 fork, originally based on PIL 1.1.7)
    version      2.0.0 (Pillow)
    platform     darwin 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
                 [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
    --- TKINTER support available
    *** JPEG support available
    --- ZLIB (PNG/ZIP) support available
    *** TIFF G3/G4 (experimental) support not available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    *** WEBP support not available
    To add a missing option, make sure you have the required
    library, and set the corresponding ROOT variable in the script.

    To check the build, run the script.

    changing mode of build/scripts-2.7/ from 644 to 755
    changing mode of build/scripts-2.7/ from 644 to 755
    changing mode of build/scripts-2.7/ from 644 to 755
    changing mode of build/scripts-2.7/ from 644 to 755
    changing mode of build/scripts-2.7/ from 644 to 755

    warning: no previously-included files found matching '.hgignore'
    warning: no previously-included files found matching '.hgtags'
    warning: no previously-included files found matching 'BUILDME.bat'
    warning: no previously-included files found matching ''
    warning: no previously-included files found matching 'SHIP'
    warning: no previously-included files found matching 'SHIP.bat'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no files found matching 'README' under directory 'docs'
    warning: no files found matching 'CHANGES' under directory 'docs'
    warning: no files found matching 'CONTENTS' under directory 'docs'
    changing mode of .../ENV-PIL/bin/ to 755
    changing mode of .../ENV-PIL/bin/ to 755
    changing mode of .../ENV-PIL/bin/ to 755
    changing mode of .../ENV-PIL/bin/ to 755
    changing mode of .../ENV-PIL/bin/ to 755
Successfully installed Pillow
Cleaning up...

That helps me with running Django on my dev server, so that I can upload JPEGs properly. But if I use PIL to draw directly, I find the output looks very grainy (e.g. from the code below), and the colours are feathered.  I have tried saving as both PNG and GIF.

img ='RGB', (500,500), (255, 255, 255))
draw = ImageDraw.Draw(img)
draw.ellipse((x-r, y-r, x+r, y+r))
font = ImageFont.truetype("examplefont.tff", 25)
w, h = draw.textsize(msg)
draw.text((x-w/2, y-h/2), msg, (0, 0, 0), font=font), fmt)

So I am happy to have Django working with images nicely, but I will stick to client-side drawing as much as possible.


Dynamic web pages with Django, Ajax and jQuery

How do you update a page’s content on the fly?

Of course, the simple answer is to use ajax.

But there are many ways to implement ajax, and sometimes it isn’t necessary. My aim here is to describe some solutions, and when each is appropriate.

Specifically, I’ll deal with sites using jQuery, and written with Django (and Django-CMS in particular, but that’s not critical to this discussion). I will also show how I hooked up Twitter Bootstrap‘s modal windows to do what I needed.

These are the approaches I have found for dynamically updating your webpage. They all have their uses:

  • Toggle hidden content
  • Generate content on the client side
  • jQuery’s $.ajax()
  • jQuery’s $.load()
  • Dajax/Dajaxice
  • a javascript MVC framework like Angular

Researching this took me a few days and quite a bit of trial and error, so with any luck by posting I will save someone else that time.

Toggle hidden content

First, you can often do without ajax, by toggling hidden content (with either explicit javascript or a suitable Twitter Bootstrap component). Here’s a simple example:


<input type="submit" class="btn-link" name="comment"
    value="Add a comment" id="toggle-content" />
<div id="extra-content">{% include "myform.html" %}</div>


$(document).ready(function() {

This is fine if you know the potential extra content in advance, the server doesn’t need to know about it, and there’s not too much of it. It has the very strong advantage of being very fast (once the original page has loaded).

Generate content on the client side

To be honest, after implementing what seemed to be a wonderful solution with ajax, using the .load() method below, I am now rewriting my webpage to generate the content as needed in javascript. The ajax solution was just too slow, and as luck would have it, the content I need to show can be based on existing content. In fact, I hope it can be as simple as copying an element if I change the css to depend on the enclosing class.

jQuery’s $.ajax()

For some things, like telling the server that the user has finished playing a game, I am using jQuery’s $.ajax() javascript command. This combines easily with a Django view to send and receive data.

Your template will look something like this (thanks to this stack overflow post for explaining a simple way to pass the CSRF token):


{% csrf_token %}
<div id="game-board" data-board-id="{{ }}" data-done-ref="{% page_url 'game-handler' %}game-over/">...</div>


function gameOver() {
    var board = $('#game-board').attr('data-board-id');
        type: "POST",
        url: $('#game-board').attr('data-done-ref'),  // or just url: "/my-url/path/"
        data: {
            csrfmiddlewaretoken: document.getElementsByName('csrfmiddlewaretoken')[0].value,
            board: board,
            move_list: move_list.join(','),
        success: function(data) {
            alert("Congratulations! You scored: "+data);
        error: function(xhr, textStatus, errorThrown) {
            alert("Please report this error: "+errorThrown+xhr.status+xhr.responseText);

And your view will look something like this (you need to set up to point to this*):

from django.http import HttpResponseRedirect, HttpResponse
from django.http import Http404

def game_over(request):
    if request.is_ajax():
            board_pk = int(request.POST['board'])
            moves = list(map(int, request.POST['move_list'].split(',')))
        except KeyError:
            return HttpResponse('Error') # incorrect post
        # do stuff, e.g. calculate a score
        return HttpResponse(str(score))
        raise Http404

(*since I am using Django-CMS, I have actually set this up as an apphook. To do this, the admin needs to manually connect a particular page to the apphook; I also require this page to have a nominated id in the advanced settings, e.g. “game-handler”, so that I can use {% page_url "game-handler" %} in the template. I have passed this from the template to the javascript so that the javascript can reside in a static js file.)

Here I am just passing a few POST parameters to the Django view, and the Django view processes them and returns the score – very simple handling of data. Json encoding of data is also fairly easy, I believe (e.g. see the Dajaxice example below).

The downside of this and the other ajax approaches is speed: any time you interact with the server, there is the potential for it to be slow. You may want to display a loading indicator along the lines of this stack overflow post.

jQuery’s $.load()

jQuery’s $.load() javascript command is magic! It makes ajax so easy you don’t even realise you’re doing it. I am using this to replace the contents of a div with new data from the server when a button is clicked.

The html and javascript is straightforward (this example is slightly more complex than it needs to be, but is my real-world situation; it is based on this stack overflow post for reloading the content of a Twitter Bootstrap modal popup window):


<a href="my-ajax-page/load/{{ }}/" data-target="#my-modal">
   Click here


$("a[data-target=#my-modal]").click(function(event) {
    var target = $(this).attr("href");
    $("#my-modal .modal-body").load(target, function() {

With the setup above, the contents of the view referenced by my-ajax-page/load/### will be loaded into the modal dialog and displayed.

In fact using the href, data-target and data-toggle tags in a Bootstrap modal window automatically calls jQuery’s .load command, but it only seems to do it the first time; in my case, I need to load new content every time the modal is clicked, hence the explicit call above.

A few caveats I have discovered about changing contents (but double-check me on this please!) -

  • Sekizai – I use sekizai blocks for my js and css (as done by Django-CMS). If you are inserting new css and js inside the html, you may want to do it without putting them inside sekizai blocks.
  • jQuery’s $(function() {…}) – if you insert new javascript code using this function (or the equivalent .ready() method), which on a normally loaded page is automatically run when the page is ready, it will not be run.


Dajax turns the paradigm on its head, allowing you to seemingly dynamically change your page’s content from python, rather than in the javascript. I haven’t looked into it, but I presume that under the hood the python code is sending an object to the javascript, and the Dajax js library is decoding it there.

This is slightly more involved to set up, requiring several changes to your Django setup (, templates etc), and I found the documentation a little sparse.  It also took me some time to get the basic examples running; I wasn’t clear on whether I needed Dajaxice or Dajax, or which was the one to get started with; I installed Dajax first not realising I had to also follow the Dajaxice instructions (which was my bad); then I had to play with the ordering of the INSTALLED_APPS to make it work (in between django.contrib.sites and django.contrib.messages).

Once set up though, your javascript is confined only to what happens when the ajax call returns; you can just use html to call the ajax script.  And your Django code is called without need to set your own urls up; Dajax autodiscovers them so long as you put your ajax views into files named instead of

From the examples on the Dajax website, it looks like with Dajaxice you need to write your own javascript to handle the returned data, whereas with Dajax you can change page elements directly from python without writing any javascript at all. The Dajax API also allows you to assign attributes to elements, add or remove css classes, and other related functions. This stuff is not too hard to do in javascript or jQuery either though.

The key advantage of Dajax that I see is it helps maintain the separation of models and views, since the code which changes the page is in python rather than in the template.


A Dajaxice example from their website is:


<input type="text" id="text"/>
<input type="button"
    callback, {'text':$('#text').val()})"


function callback(data){

With the file (in a Django app named “examples”, as referenced by the “onclick” function above):

from django.utils import simplejson
from dajaxice.decorators import dajaxice_register

def args_example(request, text):
    return simplejson.dumps({'message':'Message is %s!' % text})


And a Dajax example from the website, with one minor modification to make it clear that you do not need to write any javascript:


<input type="text" value="5" id="a"> x
<input type="text" value="6" id="b"> =
<input type="text" value="" id="result">
<input type="button" value="Multiply!"

With the file (in a Django app named examples, as referenced by the onclick method above):

from dajax.core import Dajax
from dajaxice.decorators import dajaxice_register

def multiply(request, a, b):
    dajax = Dajax()
    result = int(a) * int(b)
    return dajax.json()

A javascript MVC framework like Angular

For web apps of any complexity, it is worth going with an MVC framework so you don’t have to manage the DOM updates yourself after the ajax call.  See this post for details of how I am getting Django to work with Angular.


All these approaches have their place. If you know the potential extra content in advance and the server doesn’t need to know about it, you can toggle hidden content. If you can generate it on the client side, that will probably be more responsive than using ajax. If you want to tell the server something but don’t need to change the content, you can use .ajax(). If you need to load in a whole new section, you can use .load() (but be aware it could be slow communicating with the server).

I think Dajaxice’s functionality is covered by .ajax(), but Dajaxice nicely handles the CSRF token for you and looks clearer. Dajax would be handy if several elements need to be changed and lets you write the page-changing code in python instead of javascript. It should thereby help maintain a cleaner separation of models and views.

I’m just learning this myself, so if I’ve missed another approach or said something misleading, please let me know!

Subclassing in Django to preserve reusability

I am developing a Django app called discuss, which allows users to post comments.

Now my particular application is to allow users to comment on a game, and I would like them to be able to load in their own game boards. However, I want to write the app as reusably as possible, so I do not want the gameboards to be part of the discuss app.

I have two models:

class Discussion(models.Model):
    name = models.CharField(max_length=60)
    def __unicode__(self):

class Post(models.Model):
    discussion = models.ForeignKey(Discussion)
    forerunner = models.ForeignKey("self", blank=True, null=True)
    author = models.ForeignKey(User)
    body = models.TextField()
    created = models.DateTimeField(auto_now_add=True)
    def __unicode__(self):
        return self.title

I have been toying with a few ways of going about this:

  • Add the (content_type, object_id, content_object) trio of fields to my Post class.  This would allow the user to associate any model with their post, but only one.  These are both undesirable features for my case. It is also messy-looking to me.
  • Add ManyToManyField(Post) to my game’s Board class, i.e. point back the other way, so that the reusable app’s Post class remains pure.  This could work except that it pollutes the Board class instead; not all boards appear in posts.
  • Add a new joining model like this:
        class PostedBoard(models.Model):
            board = models.ForeignKey(game.models.Board)
            posts = models.ManyToManyField(discuss.models.Post)

    This would probably work but feels very wrong.

  • Subclass Post for the game, e.g.:
        class GamePost(discuss.models.Post):
            boards = models.ManyToManyField(game.models.Board)

The last feels like the right object-oriented approach, but I wasn’t sure how well it would actually work in Django. The purpose of this post is simple: subclassing Django models does work, with a caveat: the usual object manager does not know the new post’s subclass. This means if you use discussion.post_set.objects, you will not know the subclasses of the returned objects.

d = Discussion.objects.get()
# <Discussion: MyDiscussion>
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>]
g = GamePost(discussion = d, title = "Test subclassing", ...)  # Note that the subclass's primary key is 4, not 1
# 4
d.post_set.all() # success!
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>, <Post: Test subclassing>]
# Ah - but here's the rub: 
#      this command does not know the new post's subclass
gg = d.post_set.all()[3]
isinstance(gg, GamePost)
# False

There are a number of solutions out there to deal with this problem – this one seems well-regarded.

I have decided to go with a simple approach which takes advantage of the fact that the object’s primary key is the same whatever class it shows up as. If I need to use the instance as a member of its subclass, just use:

    def as_subclass(instance, subclass):
            return subclass.objects.get(
        except subclass.DoesNotExist:
            return instance

or, if you have lots of subclassing going on, here is a more automated method which searches through all the possible subclasses and checks each one in turn (assuming only one level of inheritance):

    def as_leaf_class(instance):
        subclasses = instance.__class__.__subclasses__()
        for subclass in subclasses:
            t = subclass.objects.filter(
            if len(t)>0: return t[0]
        return instance

I would love to hear if you’ve had a similar problem before and how you solved it!


Quick guide to creating reusable Django apps

I have developed several Django-CMS sites, e.g. Racing Tadpole, U R The Event Manager, Aquizzical and School Circle, and find myself frequently wanting to reuse pieces of the code.  Of course, this is why Django has the concept of reusable apps, and has a good intro tutorial here.

I am going to embellish on that tutorial and show the actual steps I followed to go from no app, to a single package (possibly containing multiple apps) on the Python Package Index, pypi, that anyone can install with pip, that can be found by others on djangopackages, and with source code on github.  I have already followed this process for cmsplugin-rt, which (in my humble opinion) contains lots of helpful Django-CMS plugins (particularly for Twitter Bootstrap) like buttons and a navbar, but also more generic plugins like Google fonts, Google analytics, Facebook and Twitter buttons.  You can install this easily by typing:

pip install cmsplugin-rt

I now want to repeat that process for a new set of apps, which will include some global settings for Bootstrap.

  1. Create a dummy Django project with all the apps you are going to need (e.g. Django-CMS in my case). 
  2. Write your app, starting with python startapp myapp 
  3. I have chosen to place this app one directory level down, i.e. cmsapp_rt/bsglobals/, so that I can put more than one app in my package (cmsapp_rt) and have them all installed by pip in one go. Note you need to put an file in the cmsapp_rt directory.
  4. Check your app works by including it in the INSTALLED_APPS in your file, and typing python runserver.
  5. My projects are in ~/Python/Projects.  My approach is to create a directory like ~/Python/MyPackages, and in it create directories for each pip package. Note as far as I can tell, pip packages like to use hyphens instead of underscores.  So I have a directory ~/Python/MyPackages/cmsapp-rt/.
  6. Following the Django tutorial, type: mv ~/Python/Projects/dummy/cmsapp_rt ~/Python/MyPackages/cmsapp-rt/.
  7. Create the README.txt or README.rstLICENSE and files in the ~/Python/MyPackages/cmsapp-rt/ directory, as per the tutorial. Add the docs directory. I also put in a CHANGES file (and add it to the file too). Note if you have any fixtures, you need to add them to the file in the same was as the templates. Feel free to base yours off mine if it helps.
  8. Create the file.  Note you can add a URL to github here, e.g.
        url = ''

    I also used

        from setuptools import setup
            find_packages packages = find_packages(),


            install_requires = [

    and most importantly, add to the arguments to setup:

            zip_safe = False,

    which forces the package to be installed as real files, not a zipped up egg. Django seems to struggle with the zipped up eggs.

  9. Commit your work to git – add a .gitignore file which contains at least:

    Make yourself a new repository on Github. Then type:

        git init
        git remote add origin
        git add --all
        git commit -a -m "Initial commit"
        git push -u origin master
  10. You can build the package with python sdist , as per the tutorial.
  11. Now it’s time to follow this guide from pypi.  The only two things you need from this are:
        python register
        python sdist bdist_wininst upload

    The guide says you need to register on the site before that first command, but you can just run the command and it will do that for you.  Also, I get the warning message below, but it doesn’t seem to matter (I have been using a Mac and a linux machine; maybe it is a problem for Windows?):

        Warning: Can't read registry to find the necessary compiler setting
        Make sure that Python modules _winreg, win32api or win32con are installed.
  12. Whenever you make changes to the code, you only need to update README, CHANGES, the version number in, re-commit it to git (and push it to github), and type:
    python sdist bdist_wininst upload
  13. At this point, you should be able to install the package using
        pip install projectname

    (or if you are only updating it, add --update on the end).

  14. With any luck, your dummy project will start to work again once you’ve done that, but this time, it will be drawing the app from your system’s packages, and any other project can do so too.  (In practice, you will want to use virtualenv for this.)
  15. Finally, let django packages know about your app.  This is very easy if you have already put your project on github – use the form here.

All done. I have followed this through and just published cmsapp-rt in this way, which you can now install with

    pip install cmsapp-rt

Please let me know if you have any suggestions or improvements!


Testing Django-CMS sites

I have been struggling for a while to get a test suite running for my Django-CMS sites, and had a breakthrough today which I hope can help others.

There were two problems.

I would get a string of errors like this:

FATAL ERROR - The following SQL query failed:
SELECT kc.constraint_name, kc.column_name, c.constraint_type
FROM information_schema.constraint_column_usage AS kc
JOIN information_schema.table_constraints AS c ON
kc.table_schema = c.table_schema AND
kc.table_name = c.table_name AND
kc.constraint_name = c.constraint_name
kc.table_schema = %s AND
kc.table_name = %s

The error was: no such table: information_schema.constraint_column_usage

This is actually a South problem. You need to add some lines to your

import sys
if 'test' in sys.argv:

Thanks to this Stackoverflow post for that solution, though it does not mention the above error. That post also points out that using a sqlite database for tests is much faster.

The second problem was this:

TemplateSyntaxError: You must enable the 'sekizai.context_processors.sekizai'
template context processor or use 'sekizai.context.SekizaiContext' 
to render your templates.

This was confusing because I was using sekizai correctly in my template.

This post pointed me in the right direction here. The problem was Django was raising an exception but I was never getting to see it – just this much less helpful message.

This Stackoverflow post explained how to enable logging of errors. I copied in the changes to, wrapping them inside the if 'test' statement.

Then when I ran

./ test

I got a much more useful error message: I had forgotten to set up a table that my template was assuming would exist. Easily fixed!

Hope that helps someone else.

My first, simple, test looks like this:

from django.test import TestCase
from django.contrib.auth.models import User
from cms.api import create_page, add_plugin
import myapp.models

def create_globals():
    The templates assume this exists
    return myapp.models.GlobalSettings.objects.create()

class FirstTest(TestCase):
    def test_template(self):
        Test the template can load
        superuser = User.objects.create_superuser('admin', '', 'admin')
        page = create_page("home", template, "en", created_by=superuser, published=True)
        page = create_page("second_test", template, "en", created_by=superuser, published=True)
        response = self.client.get('/')
        response = self.client.get('/second_test/')

I have recently discovered this book by Harry Percival, on how to incorporate testing into your Django site.  I highly recommend it.


Starting a new Django-CMS site with Webfaction

This is a list of steps I followed to start a new Django-CMS site hosted by Webfaction.  Their documentation is excellent, but there are lots of steps across a number of documents, so I hope that bringing them all together in one place helps someone.  At the very least I know it will help me next time.

Start webapps

Once you have a Webfaction account, add two site-specific webapps: a Django application and a static (no .htaccess) application, e.g. newproj_django and newproj_static.

Change the Websites section so your domain points to these two applications.  Give the django app the root URL and the static app the ending “/static“.

Delete the original Django application if you like.

You will also need a git webapp as per  You can just call it git.

Create a database on Webfaction

Create a new database for the app using the Webfaction control panel.  Adjust the settings.  Also change any other settings required.

Create your Django project

I have a cms_starter project on my local computer which I copy for this (largely created by following the steps from the Django-CMS docs).  Make sure the git origin of the new project (in .git/config) is set up like so:

url =

To rename the project, type:

cp -R cms_starter newproj
cd newproj
mv oldproj newproj

Then open and change the reference to settings to refer to newproj instead of oldproj.

Repeat with the file.

Repeat again with the file.

Make sure that when deployed, the Webfaction database is picked up in the  (I import a file into, and put into .gitignore, so that the same code can be used in dev and production.  The only difference is the file.)

Check it works by typing

./ runserver

Finally, commit these changes to git:

git add --all
git commit -a -m "Set up for newproj"

Set up SSH

As described at, you can immediately ssh into webfaction in a Mac terminal using


However you can make the process smoother by eliminating the need to type in your password every time.  If you already have keys set up in your ~/.ssh directory (if not see the above docs) type:

scp ~/.ssh/

Then ssh into your account and type (as per the docs):

cd ~
mkdir .ssh  # only needed if it does not already exist
cat ~/ >> ~/.ssh/authorized_keys
rm ~/
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod go-w $HOME

Now – this is not in the docs – you can make a shortcut (e.g. so you can type just ssh quickname) by adding the following to your local computer’s .ssh/config file. The last two lines stop the session from being timed out too soon.

Host quickname

User user
IdentityFile ~/.ssh/id_rsa
ServerAliveCountMax 3
ServerAliveInterval 10

From here it’s programmatic – use fabric

After following the below instructions several times, I have finally wised up and written them into a fabric file, which you’ll find attached at the end of this post (or find the most recent version at github).  With this, all you need to do from here is type: fab prod initialise.

[Update 12 June 2013 - I have now added the above steps for setting up SSH into the fab file as well.]

Set up the git repository on Webfaction

Ssh to the repos directory and initialise a bare repository (which must end in .git) using

git init --bare reponame.git

Following the documentation above, you also need to cd into that repo and turn on “http push” via

cd reponame.git
git config http.receivepack true

Push the local Django project to Webfaction

We are now ready to push your project up to Webfaction. In the newproj directory on your local computer:

git push origin master

This puts it into Webfaction’s git repo. Now we need to bring this into the newproj_django webapp.

cd webapps/newproj_django
rm -rf myproject
git clone ~/webapps/git/repos/reponame.git newproj

Tell Apache that we’ve changed the name of the project directory by opening apache2/conf/httpd.conf and replacing all three occurences of myproject with newproj.

Install Django-CMS

This is covered brilliantly in  Note though you do need to create a tmp directory which is only covered at the very end of the document.  My instructions below include this step.

We will install pip, create a tmp directory, then install Django-CMS into the local webapp newproj.  In any directory type:

easy_install-2.7 pip
mkdir $HOME/tmp
export TEMP=$HOME/tmp
cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-cms

Note for the last line you could also type the much simpler:

pip install django-cms ,

but this installs to a user-wide location, so will be available to all Django webapps.  This is a potential version control problem.

This is also a good time to install django-reversion if you use it (for django-cms 2.3.5 you need reversion version 1.6.6; drop the ==1.6.6 if you are using the latest version of django-cms):

cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-reversion==1.6.6

Use the right version of python

As described in, by default webfaction uses python2.6.6.  If you don’t use python2.7, you will get an error when you first type ./ : “ImportError: No module named”.  My solution to this is to add to $HOME/.bash_profile the line:

export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

Then, manually, every time you ssh into webfaction, you need to type:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"

I used to put this line into $HOME/.bash_profile as well, but the problem is you are hard-wiring into all your ssh sessions which web app to use. An example of why this is a problem: if you upgrade a pip installation in another web app, it will delete your installation from newproj_django and install the upgrade into your other project. Having been stung by this once, I am now taking the laborious approach of modifying PYTHONPATH specifically each time I log in.

And if I want to work on a different project, I log out, ssh back in again and repeat the export PYTHONPATH line with the different project.

Add to $HOME/.bash_profile the lines:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"
export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

The downside of this approach is that when you are in an ssh shell window, but you are working on another project in your webapps directory, your script for all your web apps will be accessing the packages you installed for newproj_django.  The website itself however will be using the right packages.

An alternative that I think solves that problem, but only works if you type the command “python” to run is to add to $HOME/.bash_profile the line:

alias python=python2.7

(This last line will not work if you type  ./ though.)

Set up the database

So far, you have a database, but it does not contain any tables yet.  Assuming you do not have data exported from elsewhere to import, you need to follow these instructions.

For a Django-CMS project using South, you will need to type (see the black screenshot on

python syncdb --all
python migrate --fake

You can add data, e.g. via fixtures, at this point if you want.

Set up static content

Make sure the settings point your static and media files something like (this is only one way to do it):

STATIC_ROOT = '/home/user/webapps/newproj_static/static'
STATIC_URL = '/static/static/'
MEDIA_ROOT = '/home/user/webapps/newproj_static/media'
MEDIA_URL = '/static/media/'

And set up static and media subdirectories in the newproj_static webapp:

cd $HOME/webapps/newproj_static/
mkdir media
mkdir static

Then type:

cd $HOME/webapps/newproj_django/newproj/
python collectstatic

Restart the webapp

Restart the webapp by typing:

And you should be able to see the default Django-CMS welcome page on .

All done!

If this page has been helpful to you, and you have yet to sign up with Webfaction, please consider signing up as my affiliate using this link. It will not affect your signup or pricing but I will receive a small commission if you do so.

Optional extras


If you add django-reversion to your installed apps after you have already set up the database, to fix up the database you will then need to type:

python syncdb
python migrate

Either way, once you have your database set up with the initial data you would like (which may be nothing), you will also need:

python createinitialrevisions


I have set up a very simple for deploying changes to the site which looks like this.  You can find the latest version (and even help improve it if you wish!) at github. I prefer to git add --all, git commit and git push manually.

After this file you’ll find a more involved fabfile which implement many of the steps above, so that all you need to type to get going is: fab prod initialise.

from fabric.api import *
from fabric.contrib.console import confirm
import fabric.contrib.files

git_dir = "$HOME/webapps/git/repos"
# replace username in the next path. Can't use $HOME from python.
remote_dir = '/home/username/webapps/newproj_django'
project_name = 'newproj'
code_dir = remote_dir + '/' + project_name
python_dir = remote_dir + '/lib/python2.7'
python_add_str = 'export PYTHONPATH="' + python_dir + ':$PYTHONPATH"; '

def migrate(app=''):
       fab migrate:app_name.
   If you have added a new app, you need to manually run
       python schemamigration app_name --init.
    if app:
        local("python schemamigration %s --auto" % app)
        local("python migrate %s" % app)
        local("python createinitialrevisions")

def prod():
    env.hosts = ['']  # or

def deploy(app_to_migrate=""):
To save some output text and time,
if you know only one app has changed its database
structure, you can run this with the app's name.

    fab prod deploy
    fab prod deploy:myapp
    with cd(code_dir):
        run("git pull")
        run(python_add_str + "python migrate %s" % app_to_migrate)
        run(python_add_str + "python createinitialrevisions") # only if using reversion
        run(python_add_str + "python collectstatic --noinput")

And add this code to automate the initial installation:

install_list = ['django-cms', 'django-reversion']
#  Methods for initial installation
def install_pip():
    Installs pip itself if needed.

    Usage :
        fab prod install_pip
    with settings(warn_only=True):
        run('mkdir $HOME/lib/python2.7')
        run('easy_install-2.7 pip')

def create_prod_git_repo(git_repo_name):
    Creates a new git repo on the server (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod create_prod_git_repo:project

    Requires the git webapp to have been created on the server.
    with cd(git_dir):
        run("git init --bare %s.git && cd %s.git && git config http.receivepack true" %

def add_prod_repo_as_origin_and_push(git_repo_name):
    Adds the git repo on the server as the local .git repo's origin, and pushes master to it.
    (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_prod_repo_as_origin:project

    Requires that the local .git/config has no origin yet (e.g. rename it first if it does).
    local("""echo '[remote "origin"]' >> .git/config""")
    local(r"echo '        fetch = +refs/heads/*:refs/remotes/origin/*' >> .git/config")
    local(r"echo '        url = %s:webapps/git/repos/%s.git' >> .git/config" % (env.hosts[0], git_repo_name))
    local(r"git push origin master")

def update_conf_file():
    Updates the apache httpd.conf file to point to the new project
    instead of the default 'myproject'.

    This is called as part of clone_into_project, or you can call
    separately as:  fab prod update_conf_file
    filepath = remote_dir + "/apache2/conf/httpd.conf"
    fabric.contrib.files.sed(filepath, 'myproject', project_name)

def clone_into_project(git_repo_name):
    Clones the git repo into the new webapp, deleting the default myproject project
    and updating the config file to point to the new project.
    Also adds a file to the project/project folder.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod clone_into_project:project
    repo_dir = git_dir + "/%s.git" % git_repo_name
    with cd(remote_dir):
        run('rm -rf myproject')
        run("git clone %s %s" % (repo_dir, project_name))
        run("echo 'MY_ENV=\"prod\"' > %s/%s/" % (project_name,project_name))

def add_dirs_to_static(static_webapp_name):
    Adds the "/static" and "/media" directories to the static webapp if needed,
    and deletes the default index.html.
    Also adds a project/project/static directory if there isn't one.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_dirs_to_static:static_webapp_name
    static_dir = '$HOME/webapps/%s' % static_webapp_name
    with settings(warn_only=True):
        with cd(static_dir):
            run("mkdir static && mkdir media")
            run("rm index.html")
        with cd(code_dir):
            run("mkdir %s/static" % project_name)

def pip_installs():
    Installs the necessary thirdparty apps
    into the local webapp (not globally) using pip.
    Also appends a helpful comment to .bashrc_profile.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod pip_installs
    pip = r'pip-2.7 install --install-option="--install-scripts=$PWD/bin" --install-option="--install-lib=$PWD/lib/python2.7" '
    with settings(warn_only=True):
        run("mkdir $HOME/tmp")
    with cd(remote_dir):
        for installation in install_list:
            run("export TEMP=$HOME/tmp && %s %s" % (pip, installation))
    run("echo '#%s' >> $HOME/.bash_profile" % python_add_str)

def initialise_database():
    Initialises the database to contain the tables required for
    Django-CMS with South.
    Runs syncdb --all and migrate --fake.

    Usage :
        fab prod initialise_database
    with cd(code_dir):
        run(python_add_str + "python syncdb --all")
        run(python_add_str + "python migrate --fake")

def initialise(static_webapp_name="myproj_static", git_repo_name="myproj"):
    In brief:
    Creates a git repo on the server, and fills in the django and static webapps with it.
    Initialises the database, and deploys the app.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod install

    Requires a git webapp, the database, the django webapp, the static webapp
    to have been created on the server already.
    Requires that the local .git/config has no origin yet (e.g. rename it if needed first).

    In detail:
      * Installs pip itself on the server, if needed
      * Creates a new git repo on the server (do not include the .git ending in git_repo_name)
      * Add it as the origin to the local git repo
      * Pushes the local code to the new repo
      * Clones it into the new webapp, deleting the default myproject project
      * Modifies the apache httpd.conf file to point to the new project, not myproject
      * Adds a file
      * Adds the "/static" and "/media" directories to the static webapp if needed
      * Adds a project/project/static directory if there isn't one
      * Adds a comment to .bash_profile for the python path (for the user's ref if desired)
      * Installs the necessary thirdparty apps
      * Initialises the database using South
      * Runs the Django-CMS cms check command
      * Deploys the app as normal (the git pull is redundant but harmless)

    with cd(code_dir):
        run(python_add_str + "python cms check")


Django – virtualenv and South

Following on from my last post, I have two basic Django sites up and running.

So now I want to tackle the next round of topics. Note that there’s a great list of useful python packages in pythonanywhere’s “batteries included” page.

  • virtualenv.  virtualenv is a great package for making sure your apps continue to function when you update the underlying components’ versions… e.g. I’m using some features of Django 1.4 that I know will be deprecated in Django 1.5.
  • South. I have already been frustrated when I add a new field to a table (or object), only to find Django cannot update the database tables for me. South sets out to solve this problem, and is as simple as adding another installed app to your settings file.

What follows is my record of how I got on with each, in the hope that I might help others who trip over the same pitfalls.

I found a great post that record some of the same discoveries here, covering git, virtualenv, south and fabric.  I’m already using git, and plan to use fabric as my needs grow.


I just grabbed the single file from the website, put it in my own Python directory and ran it by literally typing:

python ENV

I wasn’t sure if ENV was meant to be ENV or actually represented an environment name of my choice, so to be safe I used ENV. It turns out to be the environment name of your choice.

I discovered later that what I should have done was:

python ENV --system-site-packages

so that I could still access MySQLdb, for example, without having to install this separately into ENV.  Unfortunately I don’t see a way to retroactively do this, even for individual packages, though there is virtualenvwrapper that may give some of this functionality (e.g. see this Google groups post). I might just go straight to using virtualenvwrapper.

OK, so now I have a subdirectory named ENV in the same place I put, with subdirectories bin/, include/, and lib/.  I typed

source bin/activate

and understand that everything I do now is using this virtual environment.  I tried to deactivate it in the same way, but actually all you need to type is:


With ENV activated, I installed Django-CMS into ENV via

sudo pip install Django==1.4 django-cms south

(And because I did not do the --system-site-packages thing earlier, I also had to install MySQLdb pointing at my MAMP’s MySQL installation as follows… pip install mysql-python  . This gave the same error I found before when I tried to get this working – see this post. So I opened up ~/Python/virtualenv/ENV/build/mysql-python/site.cfg and added the line: mysql_config = /Applications/MAMP/Library/bin/mysql_config . Then I retyped pip install mysql-python, and it installed fine.)

Note you can use virtualenvwrapper at pythonanywhere using the advice here.


I added “south” to my existing project’s list of installed apps (in my Django project’s file), and typed:

python syncdb

This set up the new South data table south_migrationhistory.

Initial data

This link explains how to load initial data into your tables.

Standard data you want in your model is called a fixture in Django.  You can make a fixture by first entering data (e.g. through the admin interface), then using the dumpdata command.  E.g. To turn the data in the model myapp.Colour into a fixture, you can just do this:

python dumpdata myapp.Colour --indent 2 > myapp/fixtures/colours.json

Later, use the loaddata command to load this in.
There is actually a gotcha here – you need to specify an extension on your fixtures filename – i.e. do not just use colours. When you load the data, you can leave the extension off.


Mathematical web apps & visualisation