Tag Archives: python

Installing PIL on Mac OS

Installing PIL (the python image library) on my Mac is non-trivial. I get this error when I try to read or write a jpeg image:

IOError: encoder jpeg not available

and this error when I try to use any fonts:

ImportError: The _imagingft C module is not installed

I understand that Pillow is a much friendlier version of PIL, but even it does not help here, as its documentation simply states “Once you have installed the prerequisites” – with no further explanation of how to do that. Also, I have never used homebrew, and am not sure how it works with virtualenv, so I’d prefer not to use it.

Here are the steps I have followed, which solve both of these problems. They took a long time to discover!

  1. Uninstall any existing PIL you may have.  I’m afraid this is easier said than done. Fortunately I had installed PIL in a virtualenv, so I could just change to a new one and go from there. If you installed it using Pillow, you should be able to just pip uninstall Pillow, but I have not tried it.
  2. Install the jpeg library.  The way to do this is partly given by this stackoverflow post, though it unfortunately misses the last step, which you can find here. To summarise:
    1. Download libjpeg from http://www.ijg.org/files/jpegsrc.v8c.tar.gz
    2. Unpack it (either using the Finder or something like  tar zxvf jpegsrc.v8c.tar.gz)
    3. ./configure
    4. make
    5. sudo make install
    6. cp -r ~/Downloads/jpeg-XX/ /usr/local/jpeg
  3. Install the freetype library.  Get this from http://www.freetype.org/download.html, and follow the same procedure as for jpeg above (I didn’t do step 5 and it worked fine).  Copy it into /usr/local/freetype.  This time it will work.
  4. Install any other libraries you may want (e.g. see this list).
  5. pip install Pillow

To show this works, I get this output from the final command:

    --------------------------------------------------------------------
    SETUP SUMMARY (Pillow 2.0.0 fork, originally based on PIL 1.1.7)
    --------------------------------------------------------------------
    version      2.0.0 (Pillow)
    platform     darwin 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
                 [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
    --------------------------------------------------------------------
    --- TKINTER support available
    *** JPEG support available
    --- ZLIB (PNG/ZIP) support available
    *** TIFF G3/G4 (experimental) support not available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    *** WEBP support not available
    --------------------------------------------------------------------
    To add a missing option, make sure you have the required
    library, and set the corresponding ROOT variable in the
    setup.py script.

    To check the build, run the selftest.py script.

    changing mode of build/scripts-2.7/pilconvert.py from 644 to 755
    changing mode of build/scripts-2.7/pildriver.py from 644 to 755
    changing mode of build/scripts-2.7/pilfile.py from 644 to 755
    changing mode of build/scripts-2.7/pilfont.py from 644 to 755
    changing mode of build/scripts-2.7/pilprint.py from 644 to 755

    warning: no previously-included files found matching '.hgignore'
    warning: no previously-included files found matching '.hgtags'
    warning: no previously-included files found matching 'BUILDME.bat'
    warning: no previously-included files found matching 'make-manifest.py'
    warning: no previously-included files found matching 'SHIP'
    warning: no previously-included files found matching 'SHIP.bat'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no files found matching 'README' under directory 'docs'
    warning: no files found matching 'CHANGES' under directory 'docs'
    warning: no files found matching 'CONTENTS' under directory 'docs'
    changing mode of .../ENV-PIL/bin/pilconvert.py to 755
    changing mode of .../ENV-PIL/bin/pildriver.py to 755
    changing mode of .../ENV-PIL/bin/pilfile.py to 755
    changing mode of .../ENV-PIL/bin/pilfont.py to 755
    changing mode of .../ENV-PIL/bin/pilprint.py to 755
Successfully installed Pillow
Cleaning up...

That helps me with running Django on my dev server, so that I can upload JPEGs properly. But if I use PIL to draw directly, I find the output looks very grainy (e.g. from the code below), and the colours are feathered.  I have tried saving as both PNG and GIF.

img = Image.new('RGB', (500,500), (255, 255, 255))
draw = ImageDraw.Draw(img)
draw.ellipse((x-r, y-r, x+r, y+r))
font = ImageFont.truetype("examplefont.tff", 25)
w, h = draw.textsize(msg)
draw.text((x-w/2, y-h/2), msg, (0, 0, 0), font=font)
img.save(filename, fmt)

So I am happy to have Django working with images nicely, but I will stick to client-side drawing as much as possible.

  

Dynamic web pages with Django, Ajax and jQuery

How do you update a page’s content on the fly?

Of course, the simple answer is to use ajax.

But there are many ways to implement ajax, and sometimes it isn’t necessary. My aim here is to describe some solutions, and when each is appropriate.

Specifically, I’ll deal with sites using jQuery, and written with Django (and Django-CMS in particular, but that’s not critical to this discussion). I will also show how I hooked up Twitter Bootstrap‘s modal windows to do what I needed.

These are the approaches I have found for dynamically updating your webpage. They all have their uses:

  • Toggle hidden content
  • Generate content on the client side
  • jQuery’s $.ajax()
  • jQuery’s $.load()
  • Dajax/Dajaxice
  • a javascript MVC framework like Angular

Researching this took me a few days and quite a bit of trial and error, so with any luck by posting I will save someone else that time.

Toggle hidden content

First, you can often do without ajax, by toggling hidden content (with either explicit javascript or a suitable Twitter Bootstrap component). Here’s a simple example:

html:

<input type="submit" class="btn-link" name="comment"
    value="Add a comment" id="toggle-content" />
<div id="extra-content">{% include "myform.html" %}</div>

javascript:

$(document).ready(function() {
    $("#extra-content").hide();
    $("#toggle-content").click(function(){
        $("#extra-content").show();
        $("#toggle-content").hide();
    });
});

This is fine if you know the potential extra content in advance, the server doesn’t need to know about it, and there’s not too much of it. It has the very strong advantage of being very fast (once the original page has loaded).

Generate content on the client side

To be honest, after implementing what seemed to be a wonderful solution with ajax, using the .load() method below, I am now rewriting my webpage to generate the content as needed in javascript. The ajax solution was just too slow, and as luck would have it, the content I need to show can be based on existing content. In fact, I hope it can be as simple as copying an element if I change the css to depend on the enclosing class.

jQuery’s $.ajax()

For some things, like telling the server that the user has finished playing a game, I am using jQuery’s $.ajax() javascript command. This combines easily with a Django view to send and receive data.

Your template will look something like this (thanks to this stack overflow post for explaining a simple way to pass the CSRF token):

html:

{% csrf_token %}
<div id="game-board" data-board-id="{{ board.pk }}" data-done-ref="{% page_url 'game-handler' %}game-over/">...</div>

javascript:

function gameOver() {
    var board = $('#game-board').attr('data-board-id');
    $.ajax({
        type: "POST",
        url: $('#game-board').attr('data-done-ref'),  // or just url: "/my-url/path/"
        data: {
            csrfmiddlewaretoken: document.getElementsByName('csrfmiddlewaretoken')[0].value,
            board: board,
            move_list: move_list.join(','),
        },
        success: function(data) {
            alert("Congratulations! You scored: "+data);
        },
        error: function(xhr, textStatus, errorThrown) {
            alert("Please report this error: "+errorThrown+xhr.status+xhr.responseText);
        }
    });
}

And your view will look something like this (you need to set up urls.py to point to this*):

view.py:

from django.http import HttpResponseRedirect, HttpResponse
from django.http import Http404

def game_over(request):
    if request.is_ajax():
        try:
            board_pk = int(request.POST['board'])
            moves = list(map(int, request.POST['move_list'].split(',')))
        except KeyError:
            return HttpResponse('Error') # incorrect post
        # do stuff, e.g. calculate a score
        return HttpResponse(str(score))
    else:
        raise Http404

(*since I am using Django-CMS, I have actually set this up as an apphook. To do this, the admin needs to manually connect a particular page to the apphook; I also require this page to have a nominated id in the advanced settings, e.g. “game-handler”, so that I can use {% page_url "game-handler" %} in the template. I have passed this from the template to the javascript so that the javascript can reside in a static js file.)

Here I am just passing a few POST parameters to the Django view, and the Django view processes them and returns the score – very simple handling of data. Json encoding of data is also fairly easy, I believe (e.g. see the Dajaxice example below).

The downside of this and the other ajax approaches is speed: any time you interact with the server, there is the potential for it to be slow. You may want to display a loading indicator along the lines of this stack overflow post.

jQuery’s $.load()

jQuery’s $.load() javascript command is magic! It makes ajax so easy you don’t even realise you’re doing it. I am using this to replace the contents of a div with new data from the server when a button is clicked.

The html and javascript is straightforward (this example is slightly more complex than it needs to be, but is my real-world situation; it is based on this stack overflow post for reloading the content of a Twitter Bootstrap modal popup window):

html:

<a href="my-ajax-page/load/{{ board.pk }}/" data-target="#my-modal">
   Click here
</a>

javascript:

$("a[data-target=#my-modal]").click(function(event) {
    event.preventDefault();
    var target = $(this).attr("href");
    $("#my-modal .modal-body").load(target, function() {
         $("#my-modal").modal("show");
    });
});

With the setup above, the contents of the view referenced by my-ajax-page/load/### will be loaded into the modal dialog and displayed.

In fact using the href, data-target and data-toggle tags in a Bootstrap modal window automatically calls jQuery’s .load command, but it only seems to do it the first time; in my case, I need to load new content every time the modal is clicked, hence the explicit call above.

A few caveats I have discovered about changing contents (but double-check me on this please!) -

  • Sekizai – I use sekizai blocks for my js and css (as done by Django-CMS). If you are inserting new css and js inside the html, you may want to do it without putting them inside sekizai blocks.
  • jQuery’s $(function() {…}) – if you insert new javascript code using this function (or the equivalent .ready() method), which on a normally loaded page is automatically run when the page is ready, it will not be run.

Dajax/Dajaxice

Dajax turns the paradigm on its head, allowing you to seemingly dynamically change your page’s content from python, rather than in the javascript. I haven’t looked into it, but I presume that under the hood the python code is sending an object to the javascript, and the Dajax js library is decoding it there.

This is slightly more involved to set up, requiring several changes to your Django setup (settings.py, templates etc), and I found the documentation a little sparse.  It also took me some time to get the basic examples running; I wasn’t clear on whether I needed Dajaxice or Dajax, or which was the one to get started with; I installed Dajax first not realising I had to also follow the Dajaxice instructions (which was my bad); then I had to play with the ordering of the INSTALLED_APPS to make it work (in between django.contrib.sites and django.contrib.messages).

Once set up though, your javascript is confined only to what happens when the ajax call returns; you can just use html to call the ajax script.  And your Django code is called without need to set your own urls up; Dajax autodiscovers them so long as you put your ajax views into files named ajax.py instead of view.py.

From the examples on the Dajax website, it looks like with Dajaxice you need to write your own javascript to handle the returned data, whereas with Dajax you can change page elements directly from python without writing any javascript at all. The Dajax API also allows you to assign attributes to elements, add or remove css classes, and other related functions. This stuff is not too hard to do in javascript or jQuery either though.

The key advantage of Dajax that I see is it helps maintain the separation of models and views, since the code which changes the page is in python rather than in the template.

Dajaxice

A Dajaxice example from their website is:

html:

<input type="text" id="text"/>
<input type="button"
    onclick="Dajaxice.examples.args_example(
    callback, {'text':$('#text').val()})"
    value="Send!"/>

javascript:

function callback(data){
    alert(data.message);
}

With the ajax.py file (in a Django app named “examples”, as referenced by the “onclick” function above):

ajax.py:

from django.utils import simplejson
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def args_example(request, text):
    return simplejson.dumps({'message':'Message is %s!' % text})

Dajax

And a Dajax example from the website, with one minor modification to make it clear that you do not need to write any javascript:

html:

<input type="text" value="5" id="a"> x
<input type="text" value="6" id="b"> =
<input type="text" value="" id="result">
<input type="button" value="Multiply!"
    onclick="Dajaxice.examples.multiply(
    Dajax.process,{'a':$('#a').val(),'b':$('#b').val()});">

With the ajax.py file (in a Django app named examples, as referenced by the onclick method above):

ajax.py:

from dajax.core import Dajax
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def multiply(request, a, b):
    dajax = Dajax()
    result = int(a) * int(b)
    dajax.assign('#result','value',str(result))
    return dajax.json()

A javascript MVC framework like Angular

For web apps of any complexity, it is worth going with an MVC framework so you don’t have to manage the DOM updates yourself after the ajax call.  See this post for details of how I am getting Django to work with Angular.

Summary

All these approaches have their place. If you know the potential extra content in advance and the server doesn’t need to know about it, you can toggle hidden content. If you can generate it on the client side, that will probably be more responsive than using ajax. If you want to tell the server something but don’t need to change the content, you can use .ajax(). If you need to load in a whole new section, you can use .load() (but be aware it could be slow communicating with the server).

I think Dajaxice’s functionality is covered by .ajax(), but Dajaxice nicely handles the CSRF token for you and looks clearer. Dajax would be handy if several elements need to be changed and lets you write the page-changing code in python instead of javascript. It should thereby help maintain a cleaner separation of models and views.

I’m just learning this myself, so if I’ve missed another approach or said something misleading, please let me know!

Subclassing in Django to preserve reusability

I am developing a Django app called discuss, which allows users to post comments.

Now my particular application is to allow users to comment on a game, and I would like them to be able to load in their own game boards. However, I want to write the app as reusably as possible, so I do not want the gameboards to be part of the discuss app.

I have two models:

class Discussion(models.Model):
    name = models.CharField(max_length=60)
    def __unicode__(self):
        return self.name

class Post(models.Model):
    discussion = models.ForeignKey(Discussion)
    forerunner = models.ForeignKey("self", blank=True, null=True)
    author = models.ForeignKey(User)
    body = models.TextField()
    created = models.DateTimeField(auto_now_add=True)
    def __unicode__(self):
        return self.title

I have been toying with a few ways of going about this:

  • Add the (content_type, object_id, content_object) trio of fields to my Post class.  This would allow the user to associate any model with their post, but only one.  These are both undesirable features for my case. It is also messy-looking to me.
  • Add ManyToManyField(Post) to my game’s Board class, i.e. point back the other way, so that the reusable app’s Post class remains pure.  This could work except that it pollutes the Board class instead; not all boards appear in posts.
  • Add a new joining model like this:
        class PostedBoard(models.Model):
            board = models.ForeignKey(game.models.Board)
            posts = models.ManyToManyField(discuss.models.Post)

    This would probably work but feels very wrong.

  • Subclass Post for the game, e.g.:
        class GamePost(discuss.models.Post):
            boards = models.ManyToManyField(game.models.Board)
    

The last feels like the right object-oriented approach, but I wasn’t sure how well it would actually work in Django. The purpose of this post is simple: subclassing Django models does work, with a caveat: the usual object manager does not know the new post’s subclass. This means if you use discussion.post_set.objects, you will not know the subclasses of the returned objects.

d = Discussion.objects.get()
# <Discussion: MyDiscussion>
d.post_set.all()
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>]
g = GamePost(discussion = d, title = "Test subclassing", ...)
g.save()
g.pk  # Note that the subclass's primary key is 4, not 1
# 4
d.post_set.all() # success!
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>, <Post: Test subclassing>]
# Ah - but here's the rub: 
#      this command does not know the new post's subclass
gg = d.post_set.all()[3]
isinstance(gg, GamePost)
# False

There are a number of solutions out there to deal with this problem – this one seems well-regarded.

I have decided to go with a simple approach which takes advantage of the fact that the object’s primary key is the same whatever class it shows up as. If I need to use the instance as a member of its subclass, just use:

    def as_subclass(instance, subclass):
        try:
            return subclass.objects.get(pk=instance.pk)
        except subclass.DoesNotExist:
            return instance

or, if you have lots of subclassing going on, here is a more automated method which searches through all the possible subclasses and checks each one in turn (assuming only one level of inheritance):

    def as_leaf_class(instance):
        subclasses = instance.__class__.__subclasses__()
        for subclass in subclasses:
            t = subclass.objects.filter(pk=instance.pk)
            if len(t)>0: return t[0]
        return instance

I would love to hear if you’ve had a similar problem before and how you solved it!

  

Quick guide to creating reusable Django apps

I have developed several Django-CMS sites, e.g. Racing Tadpole, U R The Event Manager, Aquizzical and School Circle, and find myself frequently wanting to reuse pieces of the code.  Of course, this is why Django has the concept of reusable apps, and has a good intro tutorial here.

I am going to embellish on that tutorial and show the actual steps I followed to go from no app, to a single package (possibly containing multiple apps) on the Python Package Index, pypi, that anyone can install with pip, that can be found by others on djangopackages, and with source code on github.  I have already followed this process for cmsplugin-rt, which (in my humble opinion) contains lots of helpful Django-CMS plugins (particularly for Twitter Bootstrap) like buttons and a navbar, but also more generic plugins like Google fonts, Google analytics, Facebook and Twitter buttons.  You can install this easily by typing:

pip install cmsplugin-rt

I now want to repeat that process for a new set of apps, which will include some global settings for Bootstrap.

  1. Create a dummy Django project with all the apps you are going to need (e.g. Django-CMS in my case). 
  2. Write your app, starting with python manage.py startapp myapp 
  3. I have chosen to place this app one directory level down, i.e. cmsapp_rt/bsglobals/, so that I can put more than one app in my package (cmsapp_rt) and have them all installed by pip in one go. Note you need to put an __init__.py file in the cmsapp_rt directory.
  4. Check your app works by including it in the INSTALLED_APPS in your settings.py file, and typing python manage.py runserver.
  5. My projects are in ~/Python/Projects.  My approach is to create a directory like ~/Python/MyPackages, and in it create directories for each pip package. Note as far as I can tell, pip packages like to use hyphens instead of underscores.  So I have a directory ~/Python/MyPackages/cmsapp-rt/.
  6. Following the Django tutorial, type: mv ~/Python/Projects/dummy/cmsapp_rt ~/Python/MyPackages/cmsapp-rt/.
  7. Create the README.txt or README.rstLICENSE and MANIFEST.in files in the ~/Python/MyPackages/cmsapp-rt/ directory, as per the tutorial. Add the docs directory. I also put in a CHANGES file (and add it to the MANIFEST.in file too). Note if you have any fixtures, you need to add them to the MANIFEST.in file in the same was as the templates. Feel free to base yours off mine if it helps.
  8. Create the setup.py file.  Note you can add a URL to github here, e.g.
        url = 'https://github.com/RacingTadpole/cmsapp-rt'

    I also used

        from setuptools import setup
        setup(...
            find_packages packages = find_packages(),
        )

    and

            install_requires = [
                'django-singleton-admin',
            ],

    and most importantly, add to the arguments to setup:

            zip_safe = False,

    which forces the package to be installed as real files, not a zipped up egg. Django seems to struggle with the zipped up eggs.

  9. Commit your work to git – add a .gitignore file which contains at least:
        dist/
        *.egg-info
        build/

    Make yourself a new repository on Github. Then type:

        git init
        git remote add origin https://github.com/user/reponame.git
        git add --all
        git commit -a -m "Initial commit"
        git push -u origin master
  10. You can build the package with python setup.py sdist , as per the tutorial.
  11. Now it’s time to follow this guide from pypi.  The only two things you need from this are:
        python setup.py register
        python setup.py sdist bdist_wininst upload

    The guide says you need to register on the site before that first command, but you can just run the command and it will do that for you.  Also, I get the warning message below, but it doesn’t seem to matter (I have been using a Mac and a linux machine; maybe it is a problem for Windows?):

        Warning: Can't read registry to find the necessary compiler setting
        Make sure that Python modules _winreg, win32api or win32con are installed.
  12. Whenever you make changes to the code, you only need to update README, CHANGES, the version number in setup.py, re-commit it to git (and push it to github), and type:
    python setup.py sdist bdist_wininst upload
  13. At this point, you should be able to install the package using
        pip install projectname

    (or if you are only updating it, add --update on the end).

  14. With any luck, your dummy project will start to work again once you’ve done that, but this time, it will be drawing the app from your system’s packages, and any other project can do so too.  (In practice, you will want to use virtualenv for this.)
  15. Finally, let django packages know about your app.  This is very easy if you have already put your project on github – use the form here.

All done. I have followed this through and just published cmsapp-rt in this way, which you can now install with

    pip install cmsapp-rt

Please let me know if you have any suggestions or improvements!

  

Starting a new Django-CMS site with Webfaction

This is a list of steps I followed to start a new Django-CMS site hosted by Webfaction.  Their documentation is excellent, but there are lots of steps across a number of documents, so I hope that bringing them all together in one place helps someone.  At the very least I know it will help me next time.

Start webapps

Once you have a Webfaction account, add two site-specific webapps: a Django application and a static (no .htaccess) application, e.g. newproj_django and newproj_static.

Change the Websites section so your domain points to these two applications.  Give the django app the root URL and the static app the ending “/static“.

Delete the original Django application if you like.

You will also need a git webapp as per http://docs.webfaction.com/software/git.html.  You can just call it git.

Create a database on Webfaction

Create a new database for the app using the Webfaction control panel.  Adjust the settings.  Also change any other settings required.

Create your Django project

I have a cms_starter project on my local computer which I copy for this (largely created by following the steps from the Django-CMS docs).  Make sure the git origin of the new project (in .git/config) is set up like so:

url = user@user.webfactional.com:webapps/git/repos/reponame.git

To rename the project, type:

cp -R cms_starter newproj
cd newproj
mv oldproj newproj

Then open manage.py and change the reference to settings to refer to newproj instead of oldproj.

Repeat with the settings.py file.

Repeat again with the wsgi.py file.

Make sure that when deployed, the Webfaction database is picked up in the settings.py.  (I import a site_settings.py file into settings.py, and put site_settings.py into .gitignore, so that the same code can be used in dev and production.  The only difference is the site_settings.py file.)

Check it works by typing

./manage.py runserver

Finally, commit these changes to git:

git add --all
git commit -a -m "Set up for newproj"

Set up SSH

As described at http://docs.webfaction.com/user-guide/access.html, you can immediately ssh into webfaction in a Mac terminal using

ssh user@user.webfactional.com

However you can make the process smoother by eliminating the need to type in your password every time.  If you already have keys set up in your ~/.ssh directory (if not see the above docs) type:

scp ~/.ssh/id_rsa.pub user@user.webfactional.com:temp_id_rsa_key.pub

Then ssh into your account and type (as per the docs):

cd ~
mkdir .ssh  # only needed if it does not already exist
cat ~/temp_id_rsa_key.pub >> ~/.ssh/authorized_keys
rm ~/temp_id_rsa_key.pub
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod go-w $HOME

Now – this is not in the docs – you can make a shortcut (e.g. so you can type just ssh quickname) by adding the following to your local computer’s .ssh/config file. The last two lines stop the session from being timed out too soon.

Host quickname

HostName user.webfactional.com
User user
IdentityFile ~/.ssh/id_rsa
ServerAliveCountMax 3
ServerAliveInterval 10

From here it’s programmatic – use fabric

After following the below instructions several times, I have finally wised up and written them into a fabric file, which you’ll find attached at the end of this post (or find the most recent version at github).  With this, all you need to do from here is type: fab prod initialise.

[Update 12 June 2013 - I have now added the above steps for setting up SSH into the fab file as well.]

Set up the git repository on Webfaction

Ssh to the repos directory and initialise a bare repository (which must end in .git) using

git init --bare reponame.git

Following the documentation above, you also need to cd into that repo and turn on “http push” via

cd reponame.git
git config http.receivepack true

Push the local Django project to Webfaction

We are now ready to push your project up to Webfaction. In the newproj directory on your local computer:

git push origin master

This puts it into Webfaction’s git repo. Now we need to bring this into the newproj_django webapp.

cd webapps/newproj_django
rm -rf myproject
git clone ~/webapps/git/repos/reponame.git newproj

Tell Apache that we’ve changed the name of the project directory by opening apache2/conf/httpd.conf and replacing all three occurences of myproject with newproj.

Install Django-CMS

This is covered brilliantly in http://docs.webfaction.com/software/python.html?#installing-python-packages.  Note though you do need to create a tmp directory which is only covered at the very end of the document.  My instructions below include this step.

We will install pip, create a tmp directory, then install Django-CMS into the local webapp newproj.  In any directory type:

easy_install-2.7 pip
mkdir $HOME/tmp
export TEMP=$HOME/tmp
cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-cms

Note for the last line you could also type the much simpler:

pip install django-cms ,

but this installs to a user-wide location, so will be available to all Django webapps.  This is a potential version control problem.

This is also a good time to install django-reversion if you use it (for django-cms 2.3.5 you need reversion version 1.6.6; drop the ==1.6.6 if you are using the latest version of django-cms):

cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-reversion==1.6.6

Use the right version of python

As described in http://docs.webfaction.com/software/python.html, by default webfaction uses python2.6.6.  If you don’t use python2.7, you will get an error when you first type ./manage.py : “ImportError: No module named django.core.management”.  My solution to this is to add to $HOME/.bash_profile the line:

export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

Then, manually, every time you ssh into webfaction, you need to type:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"

I used to put this line into $HOME/.bash_profile as well, but the problem is you are hard-wiring into all your ssh sessions which web app to use. An example of why this is a problem: if you upgrade a pip installation in another web app, it will delete your installation from newproj_django and install the upgrade into your other project. Having been stung by this once, I am now taking the laborious approach of modifying PYTHONPATH specifically each time I log in.

And if I want to work on a different project, I log out, ssh back in again and repeat the export PYTHONPATH line with the different project.

Add to $HOME/.bash_profile the lines:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"
export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

The downside of this approach is that when you are in an ssh shell window, but you are working on another project in your webapps directory, your manage.py script for all your web apps will be accessing the packages you installed for newproj_django.  The website itself however will be using the right packages.

An alternative that I think solves that problem, but only works if you type the command “python” to run manage.py is to add to $HOME/.bash_profile the line:

alias python=python2.7

(This last line will not work if you type  ./manage.py though.)

Set up the database

So far, you have a database, but it does not contain any tables yet.  Assuming you do not have data exported from elsewhere to import, you need to follow these instructions.

For a Django-CMS project using South, you will need to type (see the black screenshot on https://www.django-cms.org/en/documentation/):

python manage.py syncdb --all
python manage.py migrate --fake

You can add data, e.g. via fixtures, at this point if you want.

Set up static content

Make sure the settings point your static and media files something like (this is only one way to do it):

STATIC_ROOT = '/home/user/webapps/newproj_static/static'
STATIC_URL = '/static/static/'
MEDIA_ROOT = '/home/user/webapps/newproj_static/media'
MEDIA_URL = '/static/media/'

And set up static and media subdirectories in the newproj_static webapp:

cd $HOME/webapps/newproj_static/
mkdir media
mkdir static

Then type:

cd $HOME/webapps/newproj_django/newproj/
python manage.py collectstatic

Restart the webapp

Restart the webapp by typing:
../apache2/bin/restart

And you should be able to see the default Django-CMS welcome page on user.webfactional.com .

All done!

If this page has been helpful to you, and you have yet to sign up with Webfaction, please consider signing up as my affiliate using this link. It will not affect your signup or pricing but I will receive a small commission if you do so.

Optional extras

Reversion

If you add django-reversion to your installed apps after you have already set up the database, to fix up the database you will then need to type:

python manage.py syncdb
python manage.py migrate

Either way, once you have your database set up with the initial data you would like (which may be nothing), you will also need:

python manage.py createinitialrevisions

Fabric

I have set up a very simple fabfile.py for deploying changes to the site which looks like this.  You can find the latest version (and even help improve it if you wish!) at github. I prefer to git add --all, git commit and git push manually.

After this file you’ll find a more involved fabfile which implement many of the steps above, so that all you need to type to get going is: fab prod initialise.

from fabric.api import *
from fabric.contrib.console import confirm
import fabric.contrib.files

git_dir = "$HOME/webapps/git/repos"
# replace username in the next path. Can't use $HOME from python.
remote_dir = '/home/username/webapps/newproj_django'
project_name = 'newproj'
code_dir = remote_dir + '/' + project_name
python_dir = remote_dir + '/lib/python2.7'
python_add_str = 'export PYTHONPATH="' + python_dir + ':$PYTHONPATH"; '

def migrate(app=''):
"""Usage:
       fab migrate:app_name.
   If you have added a new app, you need to manually run
       python manage.py schemamigration app_name --init.
"""
    if app:
        local("python manage.py schemamigration %s --auto" % app)
        local("python manage.py migrate %s" % app)
        local("python manage.py createinitialrevisions")

def prod():
    env.hosts = ['user@user.webfactional.com']  # or user@webNNN.webfaction.com

def deploy(app_to_migrate=""):
"""
To save some output text and time,
if you know only one app has changed its database
structure, you can run this with the app's name.

Usage:
    fab prod deploy
    fab prod deploy:myapp
"""
    with cd(code_dir):
        run("git pull")
        run(python_add_str + "python manage.py migrate %s" % app_to_migrate)
        run(python_add_str + "python manage.py createinitialrevisions") # only if using reversion
        run(python_add_str + "python manage.py collectstatic --noinput")
        run("../apache2/bin/restart")

And add this code to automate the initial installation:

install_list = ['django-cms', 'django-reversion']
#
#  Methods for initial installation
#
def install_pip():
    """
    Installs pip itself if needed.

    Usage :
        fab prod install_pip
    """
    with settings(warn_only=True):
        run('mkdir $HOME/lib/python2.7')
        run('easy_install-2.7 pip')

def create_prod_git_repo(git_repo_name):
    """
    Creates a new git repo on the server (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod create_prod_git_repo:project

    Requires the git webapp to have been created on the server.
    """
    with cd(git_dir):
        run("git init --bare %s.git && cd %s.git && git config http.receivepack true" %
              (git_repo_name,git_repo_name))

def add_prod_repo_as_origin_and_push(git_repo_name):
    """
    Adds the git repo on the server as the local .git repo's origin, and pushes master to it.
    (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_prod_repo_as_origin:project

    Requires that the local .git/config has no origin yet (e.g. rename it first if it does).
    """
    local("""echo '[remote "origin"]' >> .git/config""")
    local(r"echo '        fetch = +refs/heads/*:refs/remotes/origin/*' >> .git/config")
    local(r"echo '        url = %s:webapps/git/repos/%s.git' >> .git/config" % (env.hosts[0], git_repo_name))
    local(r"git push origin master")

def update_conf_file():
    """
    Updates the apache httpd.conf file to point to the new project
    instead of the default 'myproject'.

    This is called as part of clone_into_project, or you can call
    separately as:  fab prod update_conf_file
    """
    filepath = remote_dir + "/apache2/conf/httpd.conf"
    fabric.contrib.files.sed(filepath, 'myproject', project_name)

def clone_into_project(git_repo_name):
    """
    Clones the git repo into the new webapp, deleting the default myproject project
    and updating the config file to point to the new project.
    Also adds a site_settings.py file to the project/project folder.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod clone_into_project:project
    """
    repo_dir = git_dir + "/%s.git" % git_repo_name
    with cd(remote_dir):
        run('rm -rf myproject')
        run("git clone %s %s" % (repo_dir, project_name))
        run("echo 'MY_ENV=\"prod\"' > %s/%s/site_settings.py" % (project_name,project_name))
    update_conf_file()

def add_dirs_to_static(static_webapp_name):
    """
    Adds the "/static" and "/media" directories to the static webapp if needed,
    and deletes the default index.html.
    Also adds a project/project/static directory if there isn't one.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_dirs_to_static:static_webapp_name
    """
    static_dir = '$HOME/webapps/%s' % static_webapp_name
    with settings(warn_only=True):
        with cd(static_dir):
            run("mkdir static && mkdir media")
            run("rm index.html")
        with cd(code_dir):
            run("mkdir %s/static" % project_name)

def pip_installs():
    """
    Installs the necessary thirdparty apps
    into the local webapp (not globally) using pip.
    Also appends a helpful comment to .bashrc_profile.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod pip_installs
    """
    pip = r'pip-2.7 install --install-option="--install-scripts=$PWD/bin" --install-option="--install-lib=$PWD/lib/python2.7" '
    with settings(warn_only=True):
        run("mkdir $HOME/tmp")
    with cd(remote_dir):
        for installation in install_list:
            run("export TEMP=$HOME/tmp && %s %s" % (pip, installation))
    run("echo '#%s' >> $HOME/.bash_profile" % python_add_str)

def initialise_database():
    """
    Initialises the database to contain the tables required for
    Django-CMS with South.
    Runs syncdb --all and migrate --fake.

    Usage :
        fab prod initialise_database
    """
    with cd(code_dir):
        run(python_add_str + "python manage.py syncdb --all")
        run(python_add_str + "python manage.py migrate --fake")

def initialise(static_webapp_name="myproj_static", git_repo_name="myproj"):
    """
    In brief:
    Creates a git repo on the server, and fills in the django and static webapps with it.
    Initialises the database, and deploys the app.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod install

    Requires a git webapp, the database, the django webapp, the static webapp
    to have been created on the server already.
    Requires that the local .git/config has no origin yet (e.g. rename it if needed first).

    In detail:
      * Installs pip itself on the server, if needed
      * Creates a new git repo on the server (do not include the .git ending in git_repo_name)
      * Add it as the origin to the local git repo
      * Pushes the local code to the new repo
      * Clones it into the new webapp, deleting the default myproject project
      * Modifies the apache httpd.conf file to point to the new project, not myproject
      * Adds a site_settings.py file
      * Adds the "/static" and "/media" directories to the static webapp if needed
      * Adds a project/project/static directory if there isn't one
      * Adds a comment to .bash_profile for the python path (for the user's ref if desired)
      * Installs the necessary thirdparty apps
      * Initialises the database using South
      * Runs the Django-CMS cms check command
      * Deploys the app as normal (the git pull is redundant but harmless)
    """
    install_pip()
    create_prod_git_repo(git_repo_name)
    add_prod_repo_as_origin_and_push(git_repo_name)
    clone_into_project(git_repo_name)
    add_dirs_to_static(static_webapp_name)
    pip_installs()
    initialise_database()

    with cd(code_dir):
        run(python_add_str + "python manage.py cms check")

    deploy()
  

Django with Django-CMS

I recently found myself starting to design a flexible model for the different pages of my Django website.  After an enjoyable morning thinking through how to structure the data it hit me: I was designing a content management system! Surely someone has done that before… So I turned my attention to Django-CMS.

To get a basic Django-CMS project running there is an impressive list of changes you need to make to your new project’s settings, but if you follow the tutorial it all works well, and you can get a page template of your own design up and running so that someone else can add content to it through Django’s very nice admin console.  Note – make sure you follow the tutorial for the same version of Django-CMS as you have installed!  Somehow I wound up following an older tutorial and it has caused me a lot of pain – lots of settings were subtly different.

Adding the tables

I found myself going around in circles trying to get South to put the Django-CMS tables into my pre-existing project, until I found this stackoverflow post.  The accepted answer is commenting out south from the INSTALLED_APPS, then running the usual python manage.py syncdb command, then uncommenting it and then doing a fake migration (manage.py migrate --fake).  Stuff like this makes me wonder if I’m really making my life any easier using Django… perhaps I should chuck it in and use one of the more popular frameworks? (But then I also remember how hard it was to move this blog off the front page of my WordPress site…)

Adding your own css and js

I see that Django-CMS uses some other packages, including django-sekizai to render css and javascript. I want to put my own css file in of course, and use Twitter Bootstrap too. From the django-sekizai documentation, you just need to add:

{% addtoblock "css" %}
	<link href="/media/css/stylesheet.css" rel="stylesheet" media="screen" type="text/css">
{% endaddtoblock %}

along with something similar for js.

However, there is a trick here, elucidated in this stackoverflow post.  These static files that you add must go into a differently named folder like local-static in your app directory, and you must add this to your settings.py:

STATICFILES_DIRS = (
    # Put strings here, like /home/html/static or C:/www/django/static.
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(PROJECT_PATH, "local-static"),
)

Talk about a trap for young players!

Outstanding – why is the static file directory inside the app directory after doing this, when the PROJECT_PATH appears to be one level up?

How to integrate an existing app into Django-CMS

This has been a very painful process, and I have yet to complete it. I am using the “AppHook” approach, though the documentation lists this as third out of five methods with little guidance to help you choose.

I made the following changes to my RTloginapp (which is described in this post):

Add a new file RTloginapp/cms_app.py:

from cms.app_base import CMSApp
from cms.apphook_pool import apphook_pool
from django.utils.translation import ugettext_lazy as _

class RTLoginApphook(CMSApp):
    name = _("RTLoginApphook")
    urls = ["RTloginapp.urls"]

apphook_pool.register(RTLoginApphook)

Add a new file RTloginapp/urls.py:

from django.conf.urls.defaults import patterns, include, url
import django.contrib.auth.views

urlpatterns = patterns('RTloginapp.views',

    url(r'^/?$', 'signin', name="signin"),
    url(r'^login/?$', 'signin'),
    url(r'^join/?$', 'join', {"forgot_link":"//password/reset", "activate_link":"//register"}),
    url(r'^register/?$', 'activate'),
    url(r'^signout/?$', 'signout'),

    url(r'^password/reset/?$', django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '//password/reset/done/'}),
    url(r'^password/reset/done/?$', django.contrib.auth.views.password_reset_done),
    url(r'^password/reset/(?P<uidb36>[0-9A-Za-z]+)-(?P<token>.+)/?$', django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '//password/done/'}),
    url(r'^password/done/?$', django.contrib.auth.views.password_reset_complete),
)

And then go into the admin panel and add an “accounts” page with the application (under advanced settings) set to RTLoginApphook.

Note if you muck up the urls reference, I found I needed to uninstall the apphook using:

python manage.py cms uninstall apphooks RTLoginApphook

and then go back to the admin page and hook it back up again to a page.

Note the RTloginapp templates may need to be modified if you have renamed any of the blocks, e.g. the Django-CMS tutorial uses “base_content” as the block name, where I wrote login/login.html to use “content”.

What are menus?

It took me a while to find a description of what was meant by a menu.  I think this Django-CMS documentation page gives a good enough introduction.

My “en” pages are missing!

This turned out to be due to the order of the MIDDLEWARE_CLASSES (a relic of the fact that I had originally followed an older tutorial and fixed it manually by adding cms.middleware.multilingual.MultilingualURLMiddleware to the end… it should be before the other cms.middleware classes). Not an easy error to find.

Summary

I wavered several times on whether to use Django-CMS.  It felt like many of the concepts were too heavy (what are app hooks?), there was limited documentation, and googling only sometimes solved my problems.  But in the end, I was able to push through and start using it, and I have found it very powerful.

  

Getting started with Django

For the past couple of weeks I have been developing a website using Django, a python-based framework for content management. This is after writing the site for some weeks directly in PHP using SQL, without a particular framework. At first the site started out quite simple, so that seemed sensible. As it grew I realised it would someday reach a point of unmanageability, so I have turned to a framework.

Frameworks have several advantages, including cleanly separating the underlying business logic (the model) from the way it is presented (the view); abstracting away from the SQL, instead presenting the objects represented by the SQL; making forms and form validation easier; making user registration easier; making database administration easier; making the URL scheme easier; and having lots of add-ons that make extended functionality easier.  I chose Django because I am a fan of Python.

Django has some great features which make it easy to convert an existing SQL database structure over, with hardly any changes required to the tables. The one sticking point for me has been the user registration. I hesitated for some time before tackling this, because uncharacteristically, the user model in Django 1.4 is fixed, and to extend it requires an additional one-to-one table.  Adding to the uncertainty, whatever you do now will not be supported in Django 1.5, when a more flexible user model will be released.  However, it is fairly easy to implement the extension, and I hope it will be easy to convert to Django 1.5 when the time comes.

So here I want to point out the stumbling blocks that I hit as a total beginner to Django, so that you can help see ahead of time what is involved in learning and using it. My comments come with the caveat that I have been using Django for a matter of weeks, so do not take me as any sort of authority on best practice. I would love feedback on how to do things better!

I think the best way to go is to describe (in a mini-tutorial) how I’ve built my two-phase user registration process, which uses email addresses in place of usernames. You’ll see how I implemented “Sign in” and “Join” pages, how to check a new user’s email is not already in the database, to send an email to the user’s email address when they first join, to only activate the user when they follow a link in that email, and to handle “forgot password” and “stay signed in”.  I’ve done this for a site using jQuery Mobile. Let’s begin!

App or project?

Begin with a project and an app, just like the poll app in the Django tutorial. I originally started building my user registration process as part of this app (I’ll call it pollapp), but have since realised I can separate it out as its own app, which I am calling RTloginapp.  RTloginapp can then be used for other projects where the User model may be different. (Remember to add 'RTloginapp' to the INSTALLED_APPS in settings.py.)

The points where RTloginapp and pollapp need to communicate are:

  • RTloginapp needs customised views, eg. to display the forgot password form, the user signin form etc.  Solution: All the templates are actually put in pollapp. If I was properly packaging up RTloginapp as well, I would put default templates in RTloginapp too. I’d like to have those default templates simply extend a base template so they have more chance of being actually useful… but I’m not sure what the standard practice is here – please let me know if you have an opinion.
  • In the URLs. I have put the RTloginapp URL scheme into pollapp/urls.py, not RTloginapp/urls.py.  This way the url scheme is not fixed for every project, and you can pass the views extra project-specific parameters (the next few points).
  • RTloginapp needs to display a customised first-time user join form, since different projects may need different information about the user. Solution: in urls.py, pass the join view an extra parameter that specifies the join form.
  • RTloginapp needs to save the first-time user’s information when they join. Solution: in urls.py, pass a function as a parameter to the join view. The function saves any extra information.
  • RTloginapp needs to know a few links – the “forgot password” URL and the “activate” URL. These are passed in via urls.py too. You can also optionally pass in a “return” URL which is passed to the templates if required, and any number of additional context settings.

So pollapp/urls.py will look like this when we’re done. Note I’ve added an extra question mark after the final slash in each URL (before the $), so that the final slash in the URL is optional. The official documentation seems to assume URLs will always finish with a slash.

from pollapp.forms import JoinForm
from pollapp.profile import SetJoinerProfile

urlpatterns = patterns('RTloginapp.views',

    url(r'^accounts/?$', 'signin'),
    url(r'^accounts/login/?$', 'signin'),
    url(r'^accounts/join/?$', 'join', {'JoinForm': JoinForm,
        'SetJoinerProfile': SetJoinerProfile,
        "forgot_link":"/accounts/password/reset",
        "activate_link":"/accounts/register"}),
    url(r'^accounts/register/?$', 'activate'),
    url(r'^accounts/signout/?$', 'signout'),
)

urlpatterns += patterns('your-app.views',
		# other urls
)

Django comes with a password-reset facility, which we can use by adding the following import:

    import django.contrib.auth.views

and these URLs:

    url(r'^accounts/password/reset/?$',
        django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '/accounts/password/reset/done/'}),
    url(r'^accounts/password/reset/done/?$',
        django.contrib.auth.views.password_reset_done),
    url(r'^accounts/password/reset/(?P[0-9A-Za-z]+)-(?P.+)/?$',
        django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '/accounts/password/done/'}),
    url(r'^accounts/password/done/?$',
        django.contrib.auth.views.password_reset_complete),

Extending the User model

This applies to Django 1.4. and just follows the documentation. Add this to settings.py:

AUTH_PROFILE_MODULE = 'pollapp.UserProfile'

and this to pollapp/models.py (changing the specifics to your needs! I’ve said here that users belong to a specific “circle”, and have a flag for whether they have read the terms of service):

from django.contrib.auth.models import User

class UserProfile(models.Model):
    # This field is required.
    user = models.OneToOneField(User)
    circle = models.ForeignKey(Circle)
    read_terms = models.BooleanField()

    def __unicode__(self):
        return self.user.email

from django.db.models.signals import post_save

def create_user_profile(sender, instance, created, **kwargs):
    if created:
        UserProfile.objects.create(user=instance)

post_save.connect(create_user_profile, sender=User)

# This next piece is mine - it changes the appearance of the 
# user in the admin panel (useful if you use an email address
# instead of a username, as we will do in the next section)

def user_display(self):
 return "%s %s (%s)" % (self.first_name, self.last_name, self.email)

User.__unicode__ = user_display

And create pollapp/admin.py:

#
# to extend the user model in the admin, as per
# https://docs.djangoproject.com/en/dev/topics/auth/
#
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth.models import User

# Define an inline admin descriptor for UserProfile model
# which acts a bit like a singleton
class UserProfileInline(admin.StackedInline):
    model = models.UserProfile
    can_delete = False
    verbose_name_plural = 'profile'

# Define a new User admin
class UserAdmin(UserAdmin):
    inlines = (UserProfileInline, )

# Re-register UserAdmin
admin.site.unregister(User)
admin.site.register(User, UserAdmin)

While we’re on this subject, we might as well define the function SetJoinerProfile alluded to before (I’ve put this in its own pollapp/profile.py file, hence the earlier from pollapp.profile import SetJoinerProfile):

from circleapp import models

def SetJoinerProfile(new_user, form):
    """
    An extra function required by RTloginapp,
    to set additional user profile data on first joining.
    The function is provided with the new user object and the JoinForm.
    It should finish by saving the user profile.
    """
    new_user_profile = new_user.get_profile()
    circle = form.cleaned_data["circle"]
    new_user_profile.cricle = circle
    new_user_profile.read_terms = form.cleaned_data['read_terms']
    new_user_profile.save()  # is this the standard approach?
    return

Note the last line is to save the new user’s profile. I haven’t found any reference online to needing to do this – I would have thought saving the user would automatically save the profile too (the user is saved by RTloginapp immediately after calling the above function). But I find that it’s not saved if you don’t explicitly save the profile.

Using an email address instead of a username

I personally like using the email address instead of a username – it’s one less thing to have to remember. But Django 1.4 needs a unique username (max of 30 chars, so I can’t really use the email address as the username). My solution is simply to hash the email address and use the first 30 characters as the username. I include a check for hash collisions, to be on the safe side, though I’m sure it’s overkill.

To make this easier, I’ve added a “backend” so that you can look up a user based on their email address, following this post.  This will allow us later to authenticate users with the command:

   user = authenticate(username=email, password=password)

For this you need to add to settings.py:

AUTHENTICATION_BACKENDS = (
    'RTloginapp.accounts.backends.EmailUsernameBackend',
    'django.contrib.auth.backends.ModelBackend'
)

I then added an accounts directory under RTloginapp, with a blank __init__.py file, and a file backends.py which contains:

from django.conf import settings
from django.contrib.auth.models import User
from RTloginapp import utils

class EmailUsernameBackend(object):
    def authenticate(self, username=None, password=None):
        """
        If the username is an email address,
        then get the user with that email address.
        Otherwise get the user with that username.
        """
        if '@' in username:
            (uname, user) = utils.get_username_and_user(email=username)
            # if that couldn't find the user, it's possible the user's
            # username was entered manually, so just look up the email address
            if user is None:
                user = utils.get_object_or_none(User, email=username)
        else:
            user = utils.get_object_or_none(User, username=username)
        if user is not None:
            if user.check_password(password):
                return user
        return None

    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except User.DoesNotExist:
            return None

This uses some utility functions I put together and saved as RTloginapp/utils.py:

def get_object_or_none(model, **kwargs):
    """
    Sample usage: user = get_or_none(User, username='me')
    """
    try:
        return model.objects.get(**kwargs)
    except model.DoesNotExist:
        return None

from django.contrib.auth.models import User
import hashlib

def _get_user(username, email):
    """Used by get_username_and_user"""
    user = get_object_or_none(User, username=username)
    if user is not None:
        if user.email != email:
            # remotely possible the hashes collide
            username = 'x'+username[:29]
            (username, user) = _get_user(username, email)
    return (username, user)

def get_username_and_user(email):
    """
    Pass the user's email address, will return a tuple of the username
    (almost always the hash of the email)
    and the user object if it exists
    """
    username = hashlib.md5(email).hexdigest()[:30]
    return _get_user(username, email)

Outstanding – how do I get the user’s identification in the admin console to show as the email address, instead of the now unintelligible username?

Forms

This is a good place to introduce the JoinForm. Let’s define the minimal form required by the login app, by putting the following into RTloginapp/forms.py. Note the use of widgets to insert placeholder text in the form.

from django import forms

class BaseRTJoinForm(forms.Form):
    name = forms.CharField(max_length=100, label="Your name",
        error_messages={'required': 'Please enter your name.'},
        widget=forms.TextInput(attrs={'placeholder': 'Your name'}))
    email = forms.EmailField(label="Email address",
        error_messages={'required': 'Please enter your email address.'},
        widget=forms.TextInput(attrs={'placeholder': 'Email address'}))
    password1 = forms.CharField(label="Choose a password",
        error_messages={'required': 'Please choose a password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Choose a password'}))
    password2 = forms.CharField(label="Re-type password",
        error_messages={'required': 'Please re-type your password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Re-type password'}))

    def clean(self):
        cleaned_data = super(BaseRTJoinForm, self).clean()
        password1 = cleaned_data.get("password1")
        password2 = cleaned_data.get("password2")

        if password1 and password2 and password1!=password2:
            msg = r"Your re-typed password does not match the first one. Please type them both again!"
            self._errors["password1"] = self.error_class([msg])
            del cleaned_data["password1"]
            del cleaned_data["password2"]

        # Always return the full collection of cleaned data.
        return cleaned_data

This just follows the principles outlined in the Django documentation.

Then add the customised join form in pollapp/forms.py, eg.:

from django import forms
from RTloginapp.forms import BaseRTJoinForm
from pollapp import models

class JoinForm(BaseRTJoinForm):
    circle = forms.ModelChoiceField(queryset=models.Circle.objects.all(),
        label="Your circle",
        error_messages={'required': 'Which circle are you in?'},
        widget=forms.Select(attrs={'data-theme': 'c', 'data-inline':'true'}))
    read_terms = forms.BooleanField(label="I agree",
        error_messages={'required': 'Please agree to the terms to join.'},
        widget=forms.CheckboxInput(attrs={'data-theme': 'c', 'data-inline':'true'}))

Sending email

We also need to be able to send emails.  This is covered nicely in this stackoverflow post. Add this to utils.py:

def send_email(email_template_name, to_emails, context_dict={}):
    """
    Sends an email based on a template (e.g. login/email/join.html).
    Requires the following files:
    email_template_name.html - html email content, rendered as a template
    email_template_name.txt - plain text content of the email as a template
    email_template_name-subject.txt - a single line with the subject as a template
    email_template_name-from.txt  - a single email address stating who it is from
    to_emails can either be a single email in a string, or a list.
    """
    from django.core.mail import EmailMultiAlternatives
    from django.template.loader import get_template
    from django.template import Context

    if isinstance(to_emails, basestring):
        to_emails = [to_emails]

    subjecttext = get_template(email_template_name+'-subject.txt')
    fromtext = get_template(email_template_name+'-from.txt')
    plaintext = get_template(email_template_name+'.txt')
    htmly     = get_template(email_template_name+'.html')
    context = Context(context_dict)

    text_from    = fromtext.render(context)
    text_subject = subjecttext.render(context)
    text_content = plaintext.render(context)
    html_content = htmly.render(context)

    msg = EmailMultiAlternatives(text_subject, text_content, text_from, to_emails)
    msg.attach_alternative(html_content, "text/html")
    if text_subject and text_from and msg and to_emails:
        try:
            msg.send()
        except:
            return False
        else:
            return True
    return False

However, to actually send emails, you also need to add to settings.py (changing the host user and password to match a gmail account that you own) as explained in this stackoverflow post:

    EMAIL_USE_TLS = True
    EMAIL_HOST = 'smtp.gmail.com'
    EMAIL_HOST_USER = '@gmail.com'
    EMAIL_HOST_PASSWORD = 'example-password'
    EMAIL_PORT = 587

Outstanding – PHP lets me send an email from any user. Django only lets me send it from the host user above (it seems to ignore the “from” field in the send() call above). Is there a way to send from different email addresses in Django depending on the function, and even ones that are not real, like “do-not-reply@…”  ?

Sites

The password reset process uses Sites.

If you haven’t already, you’ll need to add your sites using python manage.py shell, along the lines of:

   
>>> from django.contrib.sites.models import Site
>>> s = Site.objects.get(pk=1)
>>> s.domain = 'mysite.com'
>>> s.name = 'My Site'
>>> s.save()
>>> s2 = Site(domain='localhost:8000', name='My local host')
>>> s2.save()
>>> s2.id

In my settings.py I also test if DEBUG is True, and set SITE_ID to 2 if so.

The views

That covers all the set up. Now we need to define the behaviour of four pages:

  • Sign in
  • Join (including sending the new user an email so they can activate their account)
  • Activate a new user when they follow the email link
  • Sign out

Let’s go through them in order:

Sign in

from django.shortcuts import get_object_or_404, render, render_to_response
from django.http import HttpResponseRedirect, HttpResponse
from django.contrib.auth import authenticate, login, logout

def signin(request, return_url='/', **kwargs):
    context = kwargs
    context['return_url'] = return_url
    email = request.POST.get('email', '')
    if email=='':
        # first time here (or no email), so display login form
        return render(request, "login/login.html", context)
    else:
        # is it a valid user? note this utilises the emailUsernameBackend
        password = request.POST.get('password', '')
        user = authenticate(username=email, password=password)
        if user is not None and user.is_active:
            # Correct password, and the user is marked "active"
            login(request, user)
            stay = request.POST.get('stay', False)
            if stay:
                request.session.set_expiry(31*24*60*60) # a month if click "stay logged in"
            else:
                request.session.set_expiry(2*60*60) # or else just two hours
            # Redirect to a success page.
            return HttpResponseRedirect(return_url)
        else:
            # Show login form with error message
            context['invalid'] = True
            return render(request, "login/login.html", context)

This clearly depends on a template called login/login.html, which we’ll come to in the next section. I’ll summarise the context for the templates there too.

Outstanding – it would be nice to set the time-limits for logging in in the settings.py file. (Alternatively they could be parameters passed in via urls.py.)

Join

from django import forms
from django.db import IntegrityError

from RTloginapp import models, utils
from RTloginapp.forms import BaseRTJoinForm

def join(request, JoinForm=BaseRTJoinForm, SetJoinerProfile=None, return_url="/", forgot_link="/accounts/password/reset", activate_link="/accounts/register", **kwargs):
    """The user is "active" if they have confirmed their email address by
       responding to the join email.
       The user is "inactive" if they have filled in the join form but not
       yet responded to the email, or if they are removed for some reason.
       An inactive user cannot sign in.
       If a user tries to join using the email address of an inactive user,
       the old user is updated, their name and related info is overwritten,
       and an email is sent out so they can become active.
    """
    context = kwargs
    context['return_url'] = return_url
    if request.method == 'POST':
        form = JoinForm(request.POST)
        context['form'] = form
        if form.is_valid():
            (username, user) = utils.get_username_and_user(form.cleaned_data['email'])
            if user is None:
                new_user = User.objects.create_user(
                                username=username,
                                password=form.cleaned_data['password1'],
                                email=form.cleaned_data['email'])
            elif user.is_active:
                # if the user has an active account,
                # send an email with a link to "forgot password" instead
                context['new_user'] = user
                context['forgot_link'] = request.build_absolute_uri(forgot_link)
                context['email_ok'] = utils.send_email("login/email/already-joined", form.cleaned_data['email'], context_dict=context)
                return render(request, "login/join-email-sent.html", context)
            else:
                # the user is inactive, so update the user instead
                new_user = user
            #
            new_user.first_name = ' '.join(form.cleaned_data['name'].split(' ')[:-1])
            new_user.last_name = form.cleaned_data['name'].split(' ')[-1]
            new_user.is_active = False
            if SetJoinerProfile is not None:
                SetJoinerProfile(new_user, form)
            new_user.save()
            #
            context['new_user'] = new_user
            context['reg_link'] = request.build_absolute_uri(activate_link + "?l="+username)
            context['email_ok'] = utils.send_email("login/email/join", new_user.email, context_dict=context)
            return render(request, "login/join-email-sent.html", context)
    else:
        form = JoinForm()
        context['form'] = form
    return render(request, "login/join.html", context)

How does that look? Please let me know if you see any security problems with it. Note I also split out the first and last names from a single name field.  This split is not perfect, so I plan to always just display the two together.

Activate

A link to this page is emailed to the user when they join, with the username appended as a GET parameter. All the page does is check the GET parameter is valid, and activate the user’s account if so. The emailed link remains valid indefinitely, which may be not be ideal.

def activate(request):
    if request.method == 'GET':
        reg_id = request.GET.get('l', '')
        if reg_id=='':
            return HttpResponseRedirect("/accounts/join")
        else:
            user = utils.get_object_or_none(User, username=reg_id)
            if user is not None:
                if not user.is_active:
                    user.is_active=True
                    user.save()
                    return render(request, "login/activated.html", {'new_user': user})
                else:
                    return render(request, "login/already-active.html", {'new_user': user})
    return HttpResponseRedirect("/accounts/join")

Signout

def signout(request, return_url="/"):
    logout(request)
    return HttpResponseRedirect(return_url)

The templates

Set up a series of website and email templates, which will have the context shown at right:

   login/
        login.html              invalid (only if True)
        join-email-sent.html    form, new_user, email_ok  (and forgot_link or reg_link)
        join.html               form
        already-active.html     new_user (*)
        activated.html          new_user (*)

    login/email/
        already-joined          form, new_user, forgot_link
        join                    form, new_user, reg_link

Each of the two email templates requires four files: .html and .txt for the content, and -from.txt and -subject.txt (these should only be a single line).

For each of the above except *, the context also includes:

                               return_url
                                any keyword args passed in by the url scheme

You will also need the reset-password templates – there are some samples available in the Django documentation:

   registration/
        password_reset_complete.html
        password_reset_confirm.html
        password_reset_done.html
        password_reset_email.html
        password_reset_form.html

This post is already ridiculously long, so I’ll just show a few of the templates I’m using. I put these in pollapp/templates/, and added this directory to settings.py‘s TEMPLATE_DIRS. The details will depend on your base template and what framework you’re using; this is for jQuery Mobile. It is a little painful to have to create all these files.

login/join.html

{% extends "base.html" %}
{% block title %}Join{% endblock %}

{% block content %}</pre>
<form action="" method="post" data-ajax="false">
{% for error in form.non_field_errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% csrf_token %}
 {% for hidden in form.hidden_fields %} {{ hidden }} {% endfor %}
 {% for field in form.visible_fields %}
 {% for error in field.errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% ifequal field.auto_id "id_read_terms" %}
 Finally... please read and agree to our
 <a href="/terms" target="readterms" data-ajax="false">terms of service</a>
 {{ field }}
 <label for="{{ field.auto_id }}">{{ field.label }}</label>
 {% else %}
 <label for="{{ field.auto_id }}">{{ field.label }}:</label>
 {{ field }}
 {% endifequal %}
 {% endfor %}
<div class="right"><a href="{{ return_url }}" data-role="button" data-theme="c" data-inline="true" data-ajax="false">Cancel </a>
 <input type="submit" name="joinBtn" value="Join" data-theme="b" data-inline="true" /></div>
</form>
<pre>{% endblock %}

This is a good place to point out a strange thing about templates – I had to use “ifequal“, not “if a==b“; the latter produced an error.

login/signin.html

{% extends "base.html" %}
{% block title %}Sign in{% endblock %}
{% block content %}
  {% if form.errors or invalid %}
  Sorry, that's not a valid username or password
  {% endif %}

  Please sign in</pre>
<form action="/accounts/login/" method="POST" data-ajax="false">
{% csrf_token %}
<fieldset data-role="fieldcontain"><label class="ui-hidden-accessible" for="email">Email</label>
 <input id="email" type="email" name="email" placeholder="Email" size="35/" data-theme="b" />
 <label class="ui-hidden-accessible" for="password">Password</label>
 <span class="left"><input id="password" type="password" name="password" placeholder="Password" data-theme="b" /></span>
 <span class="right"><input type="submit" name="login" value="OK" data-theme="b" data-inline="true" /></span></fieldset>
<fieldset><span class="left">
 <input id="stay" type="checkbox" name="stay" data-theme="a" data-inline="true" data-mini="true" />
 <label for="stay">Stay signed in</label>
 </span>
 <span class="right">
 <a href="/accounts/password/reset" data-ajax="false">Forgot password?</a>
 </span></fieldset>
</form>
<pre>{% endblock %}

login/email/join.html

Thank you for asking to join my site.
To complete your request, please click the following link:
<a href="{{ reg_link }}">{{ reg_link }}</a>
We ask this to make sure that your email address has not been used by someone else.
If you did not ask to join, please ignore this email, or
email our administrators.
Thank you
My Site

Conclusion

If you followed the above, and added a few more templates of your own, you should have a reasonably functioning user registration system. (I make no guarantees though – use it at your own risk!) In fact, if that’s what you want, I would suggest you take a look at django-registration instead, although I think that still requires a username in addition to an email address.

My real aim in writing this up, though, is to answer some of the newbie questions I had about Django and to help you avoid some of the trouble spots. In particular, how to do some very basic things, like:

  • decide what’s an app and what’s a project?
  • use an email address instead of a username for user registration?
  • extend the user model?
  • send (nicely formatted) emails?
  • build forms so that extra validation can be automatically included?
  • include extra fields like “placeholder” in forms?

I hope I’ve helped to answer all these questions in this post.  Let me know!

Also, look out for my next post on Django, which will cover my discoveries using extensions like Django-CMS, South and virtualenv.

  

Turning on the Google App Engine

When it comes to deploying a python script on the web, Google App Engine is one approach.  It took me a while to work out how to get started on this, but here is the way forward if you already have a domain site registered with Google Apps. Until I did this, I repeatedly got an “Unauthorized” message when I went to the AppEngine page.

  • log in as the admin for your site, e.g. at mail.yourdomainname.com
  • go to the control panel, www.google.com/a/yourdomainname.com
  • choose Organizations and Users, and the Services tab within that
  • turn Google App Engine on, and save changes
  • go to www.appengine.com/a/yourdomainname.com
  

Access MAMP’s MySQL database from Python

I posted a while ago about running a locally hosted server using MAMP, and I have been successfully using a PHP script on this server to read and write to a database.

Then a friend suggested I could use Python instead of PHP, e.g. using “CherryPy” as a light-weight server.  I am a big fan of python, so I want to try this.

The basic overview is:

  • Use the CherryPy python library to run the server (though more recently I have switched to Django)
  • Use the MySQLdb python library to interface with MySQL
  • Use the MAMP-installed instance of MySQL

Here’s how I made it work:

    • Download CherryPy (or Django or any other Python-based framework) from the link above.  I had to use sudo in front of the python setup.py build command.
    • Download MySQLdb from the link above.
    • Check it works by going to the terminal, typing python and then import MySQLdb
    • It doesn’t work, for several reasons:
      • It can’t find the “mysql_config” file associated with MAMP.  To fix this, open site.cfg (in the directory you installed MySQLdb) and add the line below. Note this points to the MAMP installation of my_sql, not the default of /usr/local/bin:
          mysql_config = /Applications/MAMP/Library/bin/mysql_config
      • It couldn’t find llvm. To solve this, open XCode, go to Preferences, choose the Components tab, and install the command line tools.
      • MAMP does not include the required .h files.  This post describes how to get around this: I downloaded the 64bit MySQL from here, and copied files from its include directory into the new directory MAMP/Library/include, and from its lib directory into MAMP/Library/lib. (Don’t use the 32bit files.)
      • The final magic touch is you need to type
          export DYLD_LIBRARY_PATH=/Applications/MAMP/Library/lib
        into the terminal before you go into python. (But note – this will mess up git.  So when you need git, type export DYLD_LIBRARY_PATH=''. Urrggh!)
    • Now both import MySQLdb and import _mysql work in python.
    • To access MySQL from the terminal, open MAMP and start the servers.  Then type
        /Applications/MAMP/Library/bin/mysql --host localhost -uroot -proot
      You can now do things like show databases; to look at the databases you have available.
    • So far so good, but when I try to connect to a database, I got the error “Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)” (error number 2002).  The solution is nicely explained in this post, and is:
        sudo ln -s /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock
    • Strangely, when I came back to this a week later, I got an error: “Library not loaded: libmysqlclient.18.dylib”.  I found this works:
        sudo ln -s /Applications/MAMP/Library/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib
      I also found I had to repeat the previous point to recreate the /tmp/mysql.sock file.

To summarise the on-going usage:

  • Before using python, you need to type export DYLD_LIBRARY_PATH=/Applications/MAMP/Library/lib
  • In order to use MySQL in the terminal, you need to type /Applications/MAMP/Library/bin/mysql --host -localhost -uroot -proot

In Django, you will need to use the following DATABASE values in settings.py:

    'HOST': '/Applications/MAMP/tmp/mysql/mysql.sock',
    'PORT': '8888',

One more thing. When I write apps that interface with the MAMP localhost, I need to replace localhost with the local IP address (from the Mac Network Utility program), and append port :8888.  However the CherryPy localhost has the local IP address 127.0.0.1 (and  :8080). Why are the two local IP addresses different?

From here:

Any thoughts?