Tag Archives: django

Use Django fixtures to export and import a database

I have some Django sites, and I’ve often wondered about how to make my dev database faithfully mirror the production database (at a point in time), i.e. copy all the info from my prod database back to my dev environment.

Approach 1: Export and import the data

You would think it is simple: just export the data from one and import it into the other. If you use the same type of database (e.g. MySQL) in both environments, this should work fine.  But I use sqlite in dev and MySQL in prod. Using sqlite on the development server is quick and easy, with no need to set up MySQL (see my earlier post on some of the hoops you may need to jump through for that to work). But then there are subtle differences between the import and export formats, and you can waste a lot of time mucking around with the files trying to get it to work.

Approach 2: Fixtures

So, the second idea is: use fixtures, e.g. on the production machine:

./manage.py dumpdata --indent 2 > all.json

and load it in again on the dev server (I use git to transfer it but there’s probably a cleaner approach):

./manage.py loaddata all.json

Ah – another problem. The dumpdata command, with no apps supplied, dumps everything including Users, User profiles, Sites, etc. Your dev database, even if you only just set it up a second ago, will have some content already. So you will get error messages like this:

IntegrityError: Could not load myapp.UserProfile(pk=18): column user_id is not unique

You could go back and only list the apps you want to transfer the data of after the dumpdata command, but this is painful and you may not know them all if you’re using lots of third party apps (South, Django-CMS, etc).

The solution: Fixtures + some delete statements

But I can give you good news: there is a way to do it by deleting all that initial content from your dev database, and potentially making one small deletion from the above json file.

Start with a newly created database, with the tables installed (e.g. using ./manage.py syncdb). These are the tables you need to wipe clean:

  • django_site
  • django_content_type
  • auth_permission
  • auth_user
  • south_migrationhistory if you’re using South.  Note: I am using South but did not delete the contents of this table, and it all worked. But I suspect you should delete the contents.
  • any user profiles you define (at least in Django 1.4, I’m not sure how they work in Django 1.5 yet)

In sqlite you’d do this by typing (assuming your database file is named sqlite.db):

sqlite3 sqlite.db
  delete from django_site;
  delete from django_content_type;
  delete from auth_permission;
  delete from auth_user;
  delete from south_migrationhistory;
  delete from ...;

Looks scary, but this is a brand new database anyway, right? You could easily recreate this content by starting over with ./manage.py syncdb.

The last line is where you delete your user profiles. If you do define a user profile, the act of creating a user will also create a profile. So when the users get loaded from the fixture file, user profiles will be automatically created; when the user profiles’ turn comes to be loaded from the fixture, there can be a potential clash. If you are OK to lose all your user’s profile data on your dev machine, there is a simple solution: just delete these entries from the json file.

That’s it! I hope that helps someone out there.

  

Dynamic web pages with Django, Ajax and jQuery

How do you update a page’s content on the fly?

Of course, the simple answer is to use ajax.

But there are many ways to implement ajax, and sometimes it isn’t necessary. My aim here is to describe some solutions, and when each is appropriate.

Specifically, I’ll deal with sites using jQuery, and written with Django (and Django-CMS in particular, but that’s not critical to this discussion). I will also show how I hooked up Twitter Bootstrap‘s modal windows to do what I needed.

These are the approaches I have found for dynamically updating your webpage. They all have their uses:

  • Toggle hidden content
  • Generate content on the client side
  • jQuery’s $.ajax()
  • jQuery’s $.load()
  • Dajax/Dajaxice
  • a javascript MVC framework like Angular

Researching this took me a few days and quite a bit of trial and error, so with any luck by posting I will save someone else that time.

Toggle hidden content

First, you can often do without ajax, by toggling hidden content (with either explicit javascript or a suitable Twitter Bootstrap component). Here’s a simple example:

html:

<input type="submit" class="btn-link" name="comment"
    value="Add a comment" id="toggle-content" />
<div id="extra-content">{% include "myform.html" %}</div>

javascript:

$(document).ready(function() {
    $("#extra-content").hide();
    $("#toggle-content").click(function(){
        $("#extra-content").show();
        $("#toggle-content").hide();
    });
});

This is fine if you know the potential extra content in advance, the server doesn’t need to know about it, and there’s not too much of it. It has the very strong advantage of being very fast (once the original page has loaded).

Generate content on the client side

To be honest, after implementing what seemed to be a wonderful solution with ajax, using the .load() method below, I am now rewriting my webpage to generate the content as needed in javascript. The ajax solution was just too slow, and as luck would have it, the content I need to show can be based on existing content. In fact, I hope it can be as simple as copying an element if I change the css to depend on the enclosing class.

jQuery’s $.ajax()

For some things, like telling the server that the user has finished playing a game, I am using jQuery’s $.ajax() javascript command. This combines easily with a Django view to send and receive data.

Your template will look something like this (thanks to this stack overflow post for explaining a simple way to pass the CSRF token):

html:

{% csrf_token %}
<div id="game-board" data-board-id="{{ board.pk }}" data-done-ref="{% page_url 'game-handler' %}game-over/">...</div>

javascript:

function gameOver() {
    var board = $('#game-board').attr('data-board-id');
    $.ajax({
        type: "POST",
        url: $('#game-board').attr('data-done-ref'),  // or just url: "/my-url/path/"
        data: {
            csrfmiddlewaretoken: document.getElementsByName('csrfmiddlewaretoken')[0].value,
            board: board,
            move_list: move_list.join(','),
        },
        success: function(data) {
            alert("Congratulations! You scored: "+data);
        },
        error: function(xhr, textStatus, errorThrown) {
            alert("Please report this error: "+errorThrown+xhr.status+xhr.responseText);
        }
    });
}

And your view will look something like this (you need to set up urls.py to point to this*):

view.py:

from django.http import HttpResponseRedirect, HttpResponse
from django.http import Http404

def game_over(request):
    if request.is_ajax():
        try:
            board_pk = int(request.POST['board'])
            moves = list(map(int, request.POST['move_list'].split(',')))
        except KeyError:
            return HttpResponse('Error') # incorrect post
        # do stuff, e.g. calculate a score
        return HttpResponse(str(score))
    else:
        raise Http404

(*since I am using Django-CMS, I have actually set this up as an apphook. To do this, the admin needs to manually connect a particular page to the apphook; I also require this page to have a nominated id in the advanced settings, e.g. “game-handler”, so that I can use {% page_url "game-handler" %} in the template. I have passed this from the template to the javascript so that the javascript can reside in a static js file.)

Here I am just passing a few POST parameters to the Django view, and the Django view processes them and returns the score – very simple handling of data. Json encoding of data is also fairly easy, I believe (e.g. see the Dajaxice example below).

The downside of this and the other ajax approaches is speed: any time you interact with the server, there is the potential for it to be slow. You may want to display a loading indicator along the lines of this stack overflow post.

jQuery’s $.load()

jQuery’s $.load() javascript command is magic! It makes ajax so easy you don’t even realise you’re doing it. I am using this to replace the contents of a div with new data from the server when a button is clicked.

The html and javascript is straightforward (this example is slightly more complex than it needs to be, but is my real-world situation; it is based on this stack overflow post for reloading the content of a Twitter Bootstrap modal popup window):

html:

<a href="my-ajax-page/load/{{ board.pk }}/" data-target="#my-modal">
   Click here
</a>

javascript:

$("a[data-target=#my-modal]").click(function(event) {
    event.preventDefault();
    var target = $(this).attr("href");
    $("#my-modal .modal-body").load(target, function() {
         $("#my-modal").modal("show");
    });
});

With the setup above, the contents of the view referenced by my-ajax-page/load/### will be loaded into the modal dialog and displayed.

In fact using the href, data-target and data-toggle tags in a Bootstrap modal window automatically calls jQuery’s .load command, but it only seems to do it the first time; in my case, I need to load new content every time the modal is clicked, hence the explicit call above.

A few caveats I have discovered about changing contents (but double-check me on this please!) -

  • Sekizai – I use sekizai blocks for my js and css (as done by Django-CMS). If you are inserting new css and js inside the html, you may want to do it without putting them inside sekizai blocks.
  • jQuery’s $(function() {…}) – if you insert new javascript code using this function (or the equivalent .ready() method), which on a normally loaded page is automatically run when the page is ready, it will not be run.

Dajax/Dajaxice

Dajax turns the paradigm on its head, allowing you to seemingly dynamically change your page’s content from python, rather than in the javascript. I haven’t looked into it, but I presume that under the hood the python code is sending an object to the javascript, and the Dajax js library is decoding it there.

This is slightly more involved to set up, requiring several changes to your Django setup (settings.py, templates etc), and I found the documentation a little sparse.  It also took me some time to get the basic examples running; I wasn’t clear on whether I needed Dajaxice or Dajax, or which was the one to get started with; I installed Dajax first not realising I had to also follow the Dajaxice instructions (which was my bad); then I had to play with the ordering of the INSTALLED_APPS to make it work (in between django.contrib.sites and django.contrib.messages).

Once set up though, your javascript is confined only to what happens when the ajax call returns; you can just use html to call the ajax script.  And your Django code is called without need to set your own urls up; Dajax autodiscovers them so long as you put your ajax views into files named ajax.py instead of view.py.

From the examples on the Dajax website, it looks like with Dajaxice you need to write your own javascript to handle the returned data, whereas with Dajax you can change page elements directly from python without writing any javascript at all. The Dajax API also allows you to assign attributes to elements, add or remove css classes, and other related functions. This stuff is not too hard to do in javascript or jQuery either though.

The key advantage of Dajax that I see is it helps maintain the separation of models and views, since the code which changes the page is in python rather than in the template.

Dajaxice

A Dajaxice example from their website is:

html:

<input type="text" id="text"/>
<input type="button"
    onclick="Dajaxice.examples.args_example(
    callback, {'text':$('#text').val()})"
    value="Send!"/>

javascript:

function callback(data){
    alert(data.message);
}

With the ajax.py file (in a Django app named “examples”, as referenced by the “onclick” function above):

ajax.py:

from django.utils import simplejson
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def args_example(request, text):
    return simplejson.dumps({'message':'Message is %s!' % text})

Dajax

And a Dajax example from the website, with one minor modification to make it clear that you do not need to write any javascript:

html:

<input type="text" value="5" id="a"> x
<input type="text" value="6" id="b"> =
<input type="text" value="" id="result">
<input type="button" value="Multiply!"
    onclick="Dajaxice.examples.multiply(
    Dajax.process,{'a':$('#a').val(),'b':$('#b').val()});">

With the ajax.py file (in a Django app named examples, as referenced by the onclick method above):

ajax.py:

from dajax.core import Dajax
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def multiply(request, a, b):
    dajax = Dajax()
    result = int(a) * int(b)
    dajax.assign('#result','value',str(result))
    return dajax.json()

A javascript MVC framework like Angular

For web apps of any complexity, it is worth going with an MVC framework so you don’t have to manage the DOM updates yourself after the ajax call.  See this post for details of how I am getting Django to work with Angular.

Summary

All these approaches have their place. If you know the potential extra content in advance and the server doesn’t need to know about it, you can toggle hidden content. If you can generate it on the client side, that will probably be more responsive than using ajax. If you want to tell the server something but don’t need to change the content, you can use .ajax(). If you need to load in a whole new section, you can use .load() (but be aware it could be slow communicating with the server).

I think Dajaxice’s functionality is covered by .ajax(), but Dajaxice nicely handles the CSRF token for you and looks clearer. Dajax would be handy if several elements need to be changed and lets you write the page-changing code in python instead of javascript. It should thereby help maintain a cleaner separation of models and views.

I’m just learning this myself, so if I’ve missed another approach or said something misleading, please let me know!

Subclassing in Django to preserve reusability

I am developing a Django app called discuss, which allows users to post comments.

Now my particular application is to allow users to comment on a game, and I would like them to be able to load in their own game boards. However, I want to write the app as reusably as possible, so I do not want the gameboards to be part of the discuss app.

I have two models:

class Discussion(models.Model):
    name = models.CharField(max_length=60)
    def __unicode__(self):
        return self.name

class Post(models.Model):
    discussion = models.ForeignKey(Discussion)
    forerunner = models.ForeignKey("self", blank=True, null=True)
    author = models.ForeignKey(User)
    body = models.TextField()
    created = models.DateTimeField(auto_now_add=True)
    def __unicode__(self):
        return self.title

I have been toying with a few ways of going about this:

  • Add the (content_type, object_id, content_object) trio of fields to my Post class.  This would allow the user to associate any model with their post, but only one.  These are both undesirable features for my case. It is also messy-looking to me.
  • Add ManyToManyField(Post) to my game’s Board class, i.e. point back the other way, so that the reusable app’s Post class remains pure.  This could work except that it pollutes the Board class instead; not all boards appear in posts.
  • Add a new joining model like this:
        class PostedBoard(models.Model):
            board = models.ForeignKey(game.models.Board)
            posts = models.ManyToManyField(discuss.models.Post)

    This would probably work but feels very wrong.

  • Subclass Post for the game, e.g.:
        class GamePost(discuss.models.Post):
            boards = models.ManyToManyField(game.models.Board)
    

The last feels like the right object-oriented approach, but I wasn’t sure how well it would actually work in Django. The purpose of this post is simple: subclassing Django models does work, with a caveat: the usual object manager does not know the new post’s subclass. This means if you use discussion.post_set.objects, you will not know the subclasses of the returned objects.

d = Discussion.objects.get()
# <Discussion: MyDiscussion>
d.post_set.all()
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>]
g = GamePost(discussion = d, title = "Test subclassing", ...)
g.save()
g.pk  # Note that the subclass's primary key is 4, not 1
# 4
d.post_set.all() # success!
# [<Post: First challenge>, <Post: Second challenge>, 
#  <Post: A comment on challenge 2>, <Post: Test subclassing>]
# Ah - but here's the rub: 
#      this command does not know the new post's subclass
gg = d.post_set.all()[3]
isinstance(gg, GamePost)
# False

There are a number of solutions out there to deal with this problem – this one seems well-regarded.

I have decided to go with a simple approach which takes advantage of the fact that the object’s primary key is the same whatever class it shows up as. If I need to use the instance as a member of its subclass, just use:

    def as_subclass(instance, subclass):
        try:
            return subclass.objects.get(pk=instance.pk)
        except subclass.DoesNotExist:
            return instance

or, if you have lots of subclassing going on, here is a more automated method which searches through all the possible subclasses and checks each one in turn (assuming only one level of inheritance):

    def as_leaf_class(instance):
        subclasses = instance.__class__.__subclasses__()
        for subclass in subclasses:
            t = subclass.objects.filter(pk=instance.pk)
            if len(t)>0: return t[0]
        return instance

I would love to hear if you’ve had a similar problem before and how you solved it!

  

Quick guide to creating reusable Django apps

I have developed several Django-CMS sites, e.g. Racing Tadpole, U R The Event Manager, Aquizzical and School Circle, and find myself frequently wanting to reuse pieces of the code.  Of course, this is why Django has the concept of reusable apps, and has a good intro tutorial here.

I am going to embellish on that tutorial and show the actual steps I followed to go from no app, to a single package (possibly containing multiple apps) on the Python Package Index, pypi, that anyone can install with pip, that can be found by others on djangopackages, and with source code on github.  I have already followed this process for cmsplugin-rt, which (in my humble opinion) contains lots of helpful Django-CMS plugins (particularly for Twitter Bootstrap) like buttons and a navbar, but also more generic plugins like Google fonts, Google analytics, Facebook and Twitter buttons.  You can install this easily by typing:

pip install cmsplugin-rt

I now want to repeat that process for a new set of apps, which will include some global settings for Bootstrap.

  1. Create a dummy Django project with all the apps you are going to need (e.g. Django-CMS in my case). 
  2. Write your app, starting with python manage.py startapp myapp 
  3. I have chosen to place this app one directory level down, i.e. cmsapp_rt/bsglobals/, so that I can put more than one app in my package (cmsapp_rt) and have them all installed by pip in one go. Note you need to put an __init__.py file in the cmsapp_rt directory.
  4. Check your app works by including it in the INSTALLED_APPS in your settings.py file, and typing python manage.py runserver.
  5. My projects are in ~/Python/Projects.  My approach is to create a directory like ~/Python/MyPackages, and in it create directories for each pip package. Note as far as I can tell, pip packages like to use hyphens instead of underscores.  So I have a directory ~/Python/MyPackages/cmsapp-rt/.
  6. Following the Django tutorial, type: mv ~/Python/Projects/dummy/cmsapp_rt ~/Python/MyPackages/cmsapp-rt/.
  7. Create the README.txt or README.rstLICENSE and MANIFEST.in files in the ~/Python/MyPackages/cmsapp-rt/ directory, as per the tutorial. Add the docs directory. I also put in a CHANGES file (and add it to the MANIFEST.in file too). Note if you have any fixtures, you need to add them to the MANIFEST.in file in the same was as the templates. Feel free to base yours off mine if it helps.
  8. Create the setup.py file.  Note you can add a URL to github here, e.g.
        url = 'https://github.com/RacingTadpole/cmsapp-rt'

    I also used

        from setuptools import setup
        setup(...
            find_packages packages = find_packages(),
        )

    and

            install_requires = [
                'django-singleton-admin',
            ],

    and most importantly, add to the arguments to setup:

            zip_safe = False,

    which forces the package to be installed as real files, not a zipped up egg. Django seems to struggle with the zipped up eggs.

  9. Commit your work to git – add a .gitignore file which contains at least:
        dist/
        *.egg-info
        build/

    Make yourself a new repository on Github. Then type:

        git init
        git remote add origin https://github.com/user/reponame.git
        git add --all
        git commit -a -m "Initial commit"
        git push -u origin master
  10. You can build the package with python setup.py sdist , as per the tutorial.
  11. Now it’s time to follow this guide from pypi.  The only two things you need from this are:
        python setup.py register
        python setup.py sdist bdist_wininst upload

    The guide says you need to register on the site before that first command, but you can just run the command and it will do that for you.  Also, I get the warning message below, but it doesn’t seem to matter (I have been using a Mac and a linux machine; maybe it is a problem for Windows?):

        Warning: Can't read registry to find the necessary compiler setting
        Make sure that Python modules _winreg, win32api or win32con are installed.
  12. Whenever you make changes to the code, you only need to update README, CHANGES, the version number in setup.py, re-commit it to git (and push it to github), and type:
    python setup.py sdist bdist_wininst upload
  13. At this point, you should be able to install the package using
        pip install projectname

    (or if you are only updating it, add --update on the end).

  14. With any luck, your dummy project will start to work again once you’ve done that, but this time, it will be drawing the app from your system’s packages, and any other project can do so too.  (In practice, you will want to use virtualenv for this.)
  15. Finally, let django packages know about your app.  This is very easy if you have already put your project on github – use the form here.

All done. I have followed this through and just published cmsapp-rt in this way, which you can now install with

    pip install cmsapp-rt

Please let me know if you have any suggestions or improvements!

  

Testing Django-CMS sites

I have been struggling for a while to get a test suite running for my Django-CMS sites, and had a breakthrough today which I hope can help others.

There were two problems.

I would get a string of errors like this:

FATAL ERROR - The following SQL query failed:
SELECT kc.constraint_name, kc.column_name, c.constraint_type
FROM information_schema.constraint_column_usage AS kc
JOIN information_schema.table_constraints AS c ON
kc.table_schema = c.table_schema AND
kc.table_name = c.table_name AND
kc.constraint_name = c.constraint_name
WHERE
kc.table_schema = %s AND
kc.table_name = %s

The error was: no such table: information_schema.constraint_column_usage

This is actually a South problem. You need to add some lines to your settings.py:

import sys
if 'test' in sys.argv:
    SOUTH_TESTS_MIGRATE = False

Thanks to this Stackoverflow post for that solution, though it does not mention the above error. That post also points out that using a sqlite database for tests is much faster.

The second problem was this:

TemplateSyntaxError: You must enable the 'sekizai.context_processors.sekizai'
template context processor or use 'sekizai.context.SekizaiContext' 
to render your templates.

This was confusing because I was using sekizai correctly in my template.

This post pointed me in the right direction here. The problem was Django was raising an exception but I was never getting to see it – just this much less helpful message.

This Stackoverflow post explained how to enable logging of errors. I copied in the changes to settings.py, wrapping them inside the if 'test' statement.

Then when I ran

./manage.py test

I got a much more useful error message: I had forgotten to set up a table that my template was assuming would exist. Easily fixed!

Hope that helps someone else.

My first, simple, test looks like this:

from django.test import TestCase
from django.contrib.auth.models import User
from cms.api import create_page, add_plugin
import myapp.models

def create_globals():
    """
    The templates assume this exists
    """
    return myapp.models.GlobalSettings.objects.create()

class FirstTest(TestCase):
    def test_template(self):
        """
        Test the template can load
        """
        create_globals()
        superuser = User.objects.create_superuser('admin', 'admin@admin.com', 'admin')
        template="test.html"
        page = create_page("home", template, "en", created_by=superuser, published=True)
        page = create_page("second_test", template, "en", created_by=superuser, published=True)
        response = self.client.get('/')
        response = self.client.get('/second_test/')

I have recently discovered this book by Harry Percival, on how to incorporate testing into your Django site.  I highly recommend it.

  

Starting a new Django-CMS site with Webfaction

This is a list of steps I followed to start a new Django-CMS site hosted by Webfaction.  Their documentation is excellent, but there are lots of steps across a number of documents, so I hope that bringing them all together in one place helps someone.  At the very least I know it will help me next time.

Start webapps

Once you have a Webfaction account, add two site-specific webapps: a Django application and a static (no .htaccess) application, e.g. newproj_django and newproj_static.

Change the Websites section so your domain points to these two applications.  Give the django app the root URL and the static app the ending “/static“.

Delete the original Django application if you like.

You will also need a git webapp as per http://docs.webfaction.com/software/git.html.  You can just call it git.

Create a database on Webfaction

Create a new database for the app using the Webfaction control panel.  Adjust the settings.  Also change any other settings required.

Create your Django project

I have a cms_starter project on my local computer which I copy for this (largely created by following the steps from the Django-CMS docs).  Make sure the git origin of the new project (in .git/config) is set up like so:

url = user@user.webfactional.com:webapps/git/repos/reponame.git

To rename the project, type:

cp -R cms_starter newproj
cd newproj
mv oldproj newproj

Then open manage.py and change the reference to settings to refer to newproj instead of oldproj.

Repeat with the settings.py file.

Repeat again with the wsgi.py file.

Make sure that when deployed, the Webfaction database is picked up in the settings.py.  (I import a site_settings.py file into settings.py, and put site_settings.py into .gitignore, so that the same code can be used in dev and production.  The only difference is the site_settings.py file.)

Check it works by typing

./manage.py runserver

Finally, commit these changes to git:

git add --all
git commit -a -m "Set up for newproj"

Set up SSH

As described at http://docs.webfaction.com/user-guide/access.html, you can immediately ssh into webfaction in a Mac terminal using

ssh user@user.webfactional.com

However you can make the process smoother by eliminating the need to type in your password every time.  If you already have keys set up in your ~/.ssh directory (if not see the above docs) type:

scp ~/.ssh/id_rsa.pub user@user.webfactional.com:temp_id_rsa_key.pub

Then ssh into your account and type (as per the docs):

cd ~
mkdir .ssh  # only needed if it does not already exist
cat ~/temp_id_rsa_key.pub >> ~/.ssh/authorized_keys
rm ~/temp_id_rsa_key.pub
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod go-w $HOME

Now – this is not in the docs – you can make a shortcut (e.g. so you can type just ssh quickname) by adding the following to your local computer’s .ssh/config file. The last two lines stop the session from being timed out too soon.

Host quickname

HostName user.webfactional.com
User user
IdentityFile ~/.ssh/id_rsa
ServerAliveCountMax 3
ServerAliveInterval 10

From here it’s programmatic – use fabric

After following the below instructions several times, I have finally wised up and written them into a fabric file, which you’ll find attached at the end of this post (or find the most recent version at github).  With this, all you need to do from here is type: fab prod initialise.

[Update 12 June 2013 - I have now added the above steps for setting up SSH into the fab file as well.]

Set up the git repository on Webfaction

Ssh to the repos directory and initialise a bare repository (which must end in .git) using

git init --bare reponame.git

Following the documentation above, you also need to cd into that repo and turn on “http push” via

cd reponame.git
git config http.receivepack true

Push the local Django project to Webfaction

We are now ready to push your project up to Webfaction. In the newproj directory on your local computer:

git push origin master

This puts it into Webfaction’s git repo. Now we need to bring this into the newproj_django webapp.

cd webapps/newproj_django
rm -rf myproject
git clone ~/webapps/git/repos/reponame.git newproj

Tell Apache that we’ve changed the name of the project directory by opening apache2/conf/httpd.conf and replacing all three occurences of myproject with newproj.

Install Django-CMS

This is covered brilliantly in http://docs.webfaction.com/software/python.html?#installing-python-packages.  Note though you do need to create a tmp directory which is only covered at the very end of the document.  My instructions below include this step.

We will install pip, create a tmp directory, then install Django-CMS into the local webapp newproj.  In any directory type:

easy_install-2.7 pip
mkdir $HOME/tmp
export TEMP=$HOME/tmp
cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-cms

Note for the last line you could also type the much simpler:

pip install django-cms ,

but this installs to a user-wide location, so will be available to all Django webapps.  This is a potential version control problem.

This is also a good time to install django-reversion if you use it (for django-cms 2.3.5 you need reversion version 1.6.6; drop the ==1.6.6 if you are using the latest version of django-cms):

cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-reversion==1.6.6

Use the right version of python

As described in http://docs.webfaction.com/software/python.html, by default webfaction uses python2.6.6.  If you don’t use python2.7, you will get an error when you first type ./manage.py : “ImportError: No module named django.core.management”.  My solution to this is to add to $HOME/.bash_profile the line:

export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

Then, manually, every time you ssh into webfaction, you need to type:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"

I used to put this line into $HOME/.bash_profile as well, but the problem is you are hard-wiring into all your ssh sessions which web app to use. An example of why this is a problem: if you upgrade a pip installation in another web app, it will delete your installation from newproj_django and install the upgrade into your other project. Having been stung by this once, I am now taking the laborious approach of modifying PYTHONPATH specifically each time I log in.

And if I want to work on a different project, I log out, ssh back in again and repeat the export PYTHONPATH line with the different project.

Add to $HOME/.bash_profile the lines:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"
export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

The downside of this approach is that when you are in an ssh shell window, but you are working on another project in your webapps directory, your manage.py script for all your web apps will be accessing the packages you installed for newproj_django.  The website itself however will be using the right packages.

An alternative that I think solves that problem, but only works if you type the command “python” to run manage.py is to add to $HOME/.bash_profile the line:

alias python=python2.7

(This last line will not work if you type  ./manage.py though.)

Set up the database

So far, you have a database, but it does not contain any tables yet.  Assuming you do not have data exported from elsewhere to import, you need to follow these instructions.

For a Django-CMS project using South, you will need to type (see the black screenshot on https://www.django-cms.org/en/documentation/):

python manage.py syncdb --all
python manage.py migrate --fake

You can add data, e.g. via fixtures, at this point if you want.

Set up static content

Make sure the settings point your static and media files something like (this is only one way to do it):

STATIC_ROOT = '/home/user/webapps/newproj_static/static'
STATIC_URL = '/static/static/'
MEDIA_ROOT = '/home/user/webapps/newproj_static/media'
MEDIA_URL = '/static/media/'

And set up static and media subdirectories in the newproj_static webapp:

cd $HOME/webapps/newproj_static/
mkdir media
mkdir static

Then type:

cd $HOME/webapps/newproj_django/newproj/
python manage.py collectstatic

Restart the webapp

Restart the webapp by typing:
../apache2/bin/restart

And you should be able to see the default Django-CMS welcome page on user.webfactional.com .

All done!

If this page has been helpful to you, and you have yet to sign up with Webfaction, please consider signing up as my affiliate using this link. It will not affect your signup or pricing but I will receive a small commission if you do so.

Optional extras

Reversion

If you add django-reversion to your installed apps after you have already set up the database, to fix up the database you will then need to type:

python manage.py syncdb
python manage.py migrate

Either way, once you have your database set up with the initial data you would like (which may be nothing), you will also need:

python manage.py createinitialrevisions

Fabric

I have set up a very simple fabfile.py for deploying changes to the site which looks like this.  You can find the latest version (and even help improve it if you wish!) at github. I prefer to git add --all, git commit and git push manually.

After this file you’ll find a more involved fabfile which implement many of the steps above, so that all you need to type to get going is: fab prod initialise.

from fabric.api import *
from fabric.contrib.console import confirm
import fabric.contrib.files

git_dir = "$HOME/webapps/git/repos"
# replace username in the next path. Can't use $HOME from python.
remote_dir = '/home/username/webapps/newproj_django'
project_name = 'newproj'
code_dir = remote_dir + '/' + project_name
python_dir = remote_dir + '/lib/python2.7'
python_add_str = 'export PYTHONPATH="' + python_dir + ':$PYTHONPATH"; '

def migrate(app=''):
"""Usage:
       fab migrate:app_name.
   If you have added a new app, you need to manually run
       python manage.py schemamigration app_name --init.
"""
    if app:
        local("python manage.py schemamigration %s --auto" % app)
        local("python manage.py migrate %s" % app)
        local("python manage.py createinitialrevisions")

def prod():
    env.hosts = ['user@user.webfactional.com']  # or user@webNNN.webfaction.com

def deploy(app_to_migrate=""):
"""
To save some output text and time,
if you know only one app has changed its database
structure, you can run this with the app's name.

Usage:
    fab prod deploy
    fab prod deploy:myapp
"""
    with cd(code_dir):
        run("git pull")
        run(python_add_str + "python manage.py migrate %s" % app_to_migrate)
        run(python_add_str + "python manage.py createinitialrevisions") # only if using reversion
        run(python_add_str + "python manage.py collectstatic --noinput")
        run("../apache2/bin/restart")

And add this code to automate the initial installation:

install_list = ['django-cms', 'django-reversion']
#
#  Methods for initial installation
#
def install_pip():
    """
    Installs pip itself if needed.

    Usage :
        fab prod install_pip
    """
    with settings(warn_only=True):
        run('mkdir $HOME/lib/python2.7')
        run('easy_install-2.7 pip')

def create_prod_git_repo(git_repo_name):
    """
    Creates a new git repo on the server (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod create_prod_git_repo:project

    Requires the git webapp to have been created on the server.
    """
    with cd(git_dir):
        run("git init --bare %s.git && cd %s.git && git config http.receivepack true" %
              (git_repo_name,git_repo_name))

def add_prod_repo_as_origin_and_push(git_repo_name):
    """
    Adds the git repo on the server as the local .git repo's origin, and pushes master to it.
    (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_prod_repo_as_origin:project

    Requires that the local .git/config has no origin yet (e.g. rename it first if it does).
    """
    local("""echo '[remote "origin"]' >> .git/config""")
    local(r"echo '        fetch = +refs/heads/*:refs/remotes/origin/*' >> .git/config")
    local(r"echo '        url = %s:webapps/git/repos/%s.git' >> .git/config" % (env.hosts[0], git_repo_name))
    local(r"git push origin master")

def update_conf_file():
    """
    Updates the apache httpd.conf file to point to the new project
    instead of the default 'myproject'.

    This is called as part of clone_into_project, or you can call
    separately as:  fab prod update_conf_file
    """
    filepath = remote_dir + "/apache2/conf/httpd.conf"
    fabric.contrib.files.sed(filepath, 'myproject', project_name)

def clone_into_project(git_repo_name):
    """
    Clones the git repo into the new webapp, deleting the default myproject project
    and updating the config file to point to the new project.
    Also adds a site_settings.py file to the project/project folder.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod clone_into_project:project
    """
    repo_dir = git_dir + "/%s.git" % git_repo_name
    with cd(remote_dir):
        run('rm -rf myproject')
        run("git clone %s %s" % (repo_dir, project_name))
        run("echo 'MY_ENV=\"prod\"' > %s/%s/site_settings.py" % (project_name,project_name))
    update_conf_file()

def add_dirs_to_static(static_webapp_name):
    """
    Adds the "/static" and "/media" directories to the static webapp if needed,
    and deletes the default index.html.
    Also adds a project/project/static directory if there isn't one.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_dirs_to_static:static_webapp_name
    """
    static_dir = '$HOME/webapps/%s' % static_webapp_name
    with settings(warn_only=True):
        with cd(static_dir):
            run("mkdir static && mkdir media")
            run("rm index.html")
        with cd(code_dir):
            run("mkdir %s/static" % project_name)

def pip_installs():
    """
    Installs the necessary thirdparty apps
    into the local webapp (not globally) using pip.
    Also appends a helpful comment to .bashrc_profile.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod pip_installs
    """
    pip = r'pip-2.7 install --install-option="--install-scripts=$PWD/bin" --install-option="--install-lib=$PWD/lib/python2.7" '
    with settings(warn_only=True):
        run("mkdir $HOME/tmp")
    with cd(remote_dir):
        for installation in install_list:
            run("export TEMP=$HOME/tmp && %s %s" % (pip, installation))
    run("echo '#%s' >> $HOME/.bash_profile" % python_add_str)

def initialise_database():
    """
    Initialises the database to contain the tables required for
    Django-CMS with South.
    Runs syncdb --all and migrate --fake.

    Usage :
        fab prod initialise_database
    """
    with cd(code_dir):
        run(python_add_str + "python manage.py syncdb --all")
        run(python_add_str + "python manage.py migrate --fake")

def initialise(static_webapp_name="myproj_static", git_repo_name="myproj"):
    """
    In brief:
    Creates a git repo on the server, and fills in the django and static webapps with it.
    Initialises the database, and deploys the app.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod install

    Requires a git webapp, the database, the django webapp, the static webapp
    to have been created on the server already.
    Requires that the local .git/config has no origin yet (e.g. rename it if needed first).

    In detail:
      * Installs pip itself on the server, if needed
      * Creates a new git repo on the server (do not include the .git ending in git_repo_name)
      * Add it as the origin to the local git repo
      * Pushes the local code to the new repo
      * Clones it into the new webapp, deleting the default myproject project
      * Modifies the apache httpd.conf file to point to the new project, not myproject
      * Adds a site_settings.py file
      * Adds the "/static" and "/media" directories to the static webapp if needed
      * Adds a project/project/static directory if there isn't one
      * Adds a comment to .bash_profile for the python path (for the user's ref if desired)
      * Installs the necessary thirdparty apps
      * Initialises the database using South
      * Runs the Django-CMS cms check command
      * Deploys the app as normal (the git pull is redundant but harmless)
    """
    install_pip()
    create_prod_git_repo(git_repo_name)
    add_prod_repo_as_origin_and_push(git_repo_name)
    clone_into_project(git_repo_name)
    add_dirs_to_static(static_webapp_name)
    pip_installs()
    initialise_database()

    with cd(code_dir):
        run(python_add_str + "python manage.py cms check")

    deploy()
  

Django – virtualenv and South

Following on from my last post, I have two basic Django sites up and running.

So now I want to tackle the next round of topics. Note that there’s a great list of useful python packages in pythonanywhere’s “batteries included” page.

  • virtualenv.  virtualenv is a great package for making sure your apps continue to function when you update the underlying components’ versions… e.g. I’m using some features of Django 1.4 that I know will be deprecated in Django 1.5.
  • South. I have already been frustrated when I add a new field to a table (or object), only to find Django cannot update the database tables for me. South sets out to solve this problem, and is as simple as adding another installed app to your settings file.

What follows is my record of how I got on with each, in the hope that I might help others who trip over the same pitfalls.

I found a great post that record some of the same discoveries here, covering git, virtualenv, south and fabric.  I’m already using git, and plan to use fabric as my needs grow.

VirtualEnv

I just grabbed the single file virtualenv.py from the website, put it in my own Python directory and ran it by literally typing:

python virtualenv.py ENV

I wasn’t sure if ENV was meant to be ENV or actually represented an environment name of my choice, so to be safe I used ENV. It turns out to be the environment name of your choice.

I discovered later that what I should have done was:

python virtualenv.py ENV --system-site-packages

so that I could still access MySQLdb, for example, without having to install this separately into ENV.  Unfortunately I don’t see a way to retroactively do this, even for individual packages, though there is virtualenvwrapper that may give some of this functionality (e.g. see this Google groups post). I might just go straight to using virtualenvwrapper.

OK, so now I have a subdirectory named ENV in the same place I put virtualenv.py, with subdirectories bin/, include/, and lib/.  I typed

source bin/activate

and understand that everything I do now is using this virtual environment.  I tried to deactivate it in the same way, but actually all you need to type is:

deactivate

With ENV activated, I installed Django-CMS into ENV via

sudo pip install Django==1.4 django-cms south

(And because I did not do the --system-site-packages thing earlier, I also had to install MySQLdb pointing at my MAMP’s MySQL installation as follows… pip install mysql-python  . This gave the same error I found before when I tried to get this working – see this post. So I opened up ~/Python/virtualenv/ENV/build/mysql-python/site.cfg and added the line: mysql_config = /Applications/MAMP/Library/bin/mysql_config . Then I retyped pip install mysql-python, and it installed fine.)

Note you can use virtualenvwrapper at pythonanywhere using the advice here.

South

I added “south” to my existing project’s list of installed apps (in my Django project’s settings.py file), and typed:

python manage.py syncdb

This set up the new South data table south_migrationhistory.

Initial data

This link explains how to load initial data into your tables.

Standard data you want in your model is called a fixture in Django.  You can make a fixture by first entering data (e.g. through the admin interface), then using the dumpdata command.  E.g. To turn the data in the model myapp.Colour into a fixture, you can just do this:

python manage.py dumpdata myapp.Colour --indent 2 > myapp/fixtures/colours.json

Later, use the loaddata command to load this in.
There is actually a gotcha here – you need to specify an extension on your fixtures filename – i.e. do not just use colours. When you load the data, you can leave the extension off.

  

Django with Django-CMS

I recently found myself starting to design a flexible model for the different pages of my Django website.  After an enjoyable morning thinking through how to structure the data it hit me: I was designing a content management system! Surely someone has done that before… So I turned my attention to Django-CMS.

To get a basic Django-CMS project running there is an impressive list of changes you need to make to your new project’s settings, but if you follow the tutorial it all works well, and you can get a page template of your own design up and running so that someone else can add content to it through Django’s very nice admin console.  Note – make sure you follow the tutorial for the same version of Django-CMS as you have installed!  Somehow I wound up following an older tutorial and it has caused me a lot of pain – lots of settings were subtly different.

Adding the tables

I found myself going around in circles trying to get South to put the Django-CMS tables into my pre-existing project, until I found this stackoverflow post.  The accepted answer is commenting out south from the INSTALLED_APPS, then running the usual python manage.py syncdb command, then uncommenting it and then doing a fake migration (manage.py migrate --fake).  Stuff like this makes me wonder if I’m really making my life any easier using Django… perhaps I should chuck it in and use one of the more popular frameworks? (But then I also remember how hard it was to move this blog off the front page of my WordPress site…)

Adding your own css and js

I see that Django-CMS uses some other packages, including django-sekizai to render css and javascript. I want to put my own css file in of course, and use Twitter Bootstrap too. From the django-sekizai documentation, you just need to add:

{% addtoblock "css" %}
	<link href="/media/css/stylesheet.css" rel="stylesheet" media="screen" type="text/css">
{% endaddtoblock %}

along with something similar for js.

However, there is a trick here, elucidated in this stackoverflow post.  These static files that you add must go into a differently named folder like local-static in your app directory, and you must add this to your settings.py:

STATICFILES_DIRS = (
    # Put strings here, like /home/html/static or C:/www/django/static.
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(PROJECT_PATH, "local-static"),
)

Talk about a trap for young players!

Outstanding – why is the static file directory inside the app directory after doing this, when the PROJECT_PATH appears to be one level up?

How to integrate an existing app into Django-CMS

This has been a very painful process, and I have yet to complete it. I am using the “AppHook” approach, though the documentation lists this as third out of five methods with little guidance to help you choose.

I made the following changes to my RTloginapp (which is described in this post):

Add a new file RTloginapp/cms_app.py:

from cms.app_base import CMSApp
from cms.apphook_pool import apphook_pool
from django.utils.translation import ugettext_lazy as _

class RTLoginApphook(CMSApp):
    name = _("RTLoginApphook")
    urls = ["RTloginapp.urls"]

apphook_pool.register(RTLoginApphook)

Add a new file RTloginapp/urls.py:

from django.conf.urls.defaults import patterns, include, url
import django.contrib.auth.views

urlpatterns = patterns('RTloginapp.views',

    url(r'^/?$', 'signin', name="signin"),
    url(r'^login/?$', 'signin'),
    url(r'^join/?$', 'join', {"forgot_link":"//password/reset", "activate_link":"//register"}),
    url(r'^register/?$', 'activate'),
    url(r'^signout/?$', 'signout'),

    url(r'^password/reset/?$', django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '//password/reset/done/'}),
    url(r'^password/reset/done/?$', django.contrib.auth.views.password_reset_done),
    url(r'^password/reset/(?P<uidb36>[0-9A-Za-z]+)-(?P<token>.+)/?$', django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '//password/done/'}),
    url(r'^password/done/?$', django.contrib.auth.views.password_reset_complete),
)

And then go into the admin panel and add an “accounts” page with the application (under advanced settings) set to RTLoginApphook.

Note if you muck up the urls reference, I found I needed to uninstall the apphook using:

python manage.py cms uninstall apphooks RTLoginApphook

and then go back to the admin page and hook it back up again to a page.

Note the RTloginapp templates may need to be modified if you have renamed any of the blocks, e.g. the Django-CMS tutorial uses “base_content” as the block name, where I wrote login/login.html to use “content”.

What are menus?

It took me a while to find a description of what was meant by a menu.  I think this Django-CMS documentation page gives a good enough introduction.

My “en” pages are missing!

This turned out to be due to the order of the MIDDLEWARE_CLASSES (a relic of the fact that I had originally followed an older tutorial and fixed it manually by adding cms.middleware.multilingual.MultilingualURLMiddleware to the end… it should be before the other cms.middleware classes). Not an easy error to find.

Summary

I wavered several times on whether to use Django-CMS.  It felt like many of the concepts were too heavy (what are app hooks?), there was limited documentation, and googling only sometimes solved my problems.  But in the end, I was able to push through and start using it, and I have found it very powerful.

  

Getting started with Django

For the past couple of weeks I have been developing a website using Django, a python-based framework for content management. This is after writing the site for some weeks directly in PHP using SQL, without a particular framework. At first the site started out quite simple, so that seemed sensible. As it grew I realised it would someday reach a point of unmanageability, so I have turned to a framework.

Frameworks have several advantages, including cleanly separating the underlying business logic (the model) from the way it is presented (the view); abstracting away from the SQL, instead presenting the objects represented by the SQL; making forms and form validation easier; making user registration easier; making database administration easier; making the URL scheme easier; and having lots of add-ons that make extended functionality easier.  I chose Django because I am a fan of Python.

Django has some great features which make it easy to convert an existing SQL database structure over, with hardly any changes required to the tables. The one sticking point for me has been the user registration. I hesitated for some time before tackling this, because uncharacteristically, the user model in Django 1.4 is fixed, and to extend it requires an additional one-to-one table.  Adding to the uncertainty, whatever you do now will not be supported in Django 1.5, when a more flexible user model will be released.  However, it is fairly easy to implement the extension, and I hope it will be easy to convert to Django 1.5 when the time comes.

So here I want to point out the stumbling blocks that I hit as a total beginner to Django, so that you can help see ahead of time what is involved in learning and using it. My comments come with the caveat that I have been using Django for a matter of weeks, so do not take me as any sort of authority on best practice. I would love feedback on how to do things better!

I think the best way to go is to describe (in a mini-tutorial) how I’ve built my two-phase user registration process, which uses email addresses in place of usernames. You’ll see how I implemented “Sign in” and “Join” pages, how to check a new user’s email is not already in the database, to send an email to the user’s email address when they first join, to only activate the user when they follow a link in that email, and to handle “forgot password” and “stay signed in”.  I’ve done this for a site using jQuery Mobile. Let’s begin!

App or project?

Begin with a project and an app, just like the poll app in the Django tutorial. I originally started building my user registration process as part of this app (I’ll call it pollapp), but have since realised I can separate it out as its own app, which I am calling RTloginapp.  RTloginapp can then be used for other projects where the User model may be different. (Remember to add 'RTloginapp' to the INSTALLED_APPS in settings.py.)

The points where RTloginapp and pollapp need to communicate are:

  • RTloginapp needs customised views, eg. to display the forgot password form, the user signin form etc.  Solution: All the templates are actually put in pollapp. If I was properly packaging up RTloginapp as well, I would put default templates in RTloginapp too. I’d like to have those default templates simply extend a base template so they have more chance of being actually useful… but I’m not sure what the standard practice is here – please let me know if you have an opinion.
  • In the URLs. I have put the RTloginapp URL scheme into pollapp/urls.py, not RTloginapp/urls.py.  This way the url scheme is not fixed for every project, and you can pass the views extra project-specific parameters (the next few points).
  • RTloginapp needs to display a customised first-time user join form, since different projects may need different information about the user. Solution: in urls.py, pass the join view an extra parameter that specifies the join form.
  • RTloginapp needs to save the first-time user’s information when they join. Solution: in urls.py, pass a function as a parameter to the join view. The function saves any extra information.
  • RTloginapp needs to know a few links – the “forgot password” URL and the “activate” URL. These are passed in via urls.py too. You can also optionally pass in a “return” URL which is passed to the templates if required, and any number of additional context settings.

So pollapp/urls.py will look like this when we’re done. Note I’ve added an extra question mark after the final slash in each URL (before the $), so that the final slash in the URL is optional. The official documentation seems to assume URLs will always finish with a slash.

from pollapp.forms import JoinForm
from pollapp.profile import SetJoinerProfile

urlpatterns = patterns('RTloginapp.views',

    url(r'^accounts/?$', 'signin'),
    url(r'^accounts/login/?$', 'signin'),
    url(r'^accounts/join/?$', 'join', {'JoinForm': JoinForm,
        'SetJoinerProfile': SetJoinerProfile,
        "forgot_link":"/accounts/password/reset",
        "activate_link":"/accounts/register"}),
    url(r'^accounts/register/?$', 'activate'),
    url(r'^accounts/signout/?$', 'signout'),
)

urlpatterns += patterns('your-app.views',
		# other urls
)

Django comes with a password-reset facility, which we can use by adding the following import:

    import django.contrib.auth.views

and these URLs:

    url(r'^accounts/password/reset/?$',
        django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '/accounts/password/reset/done/'}),
    url(r'^accounts/password/reset/done/?$',
        django.contrib.auth.views.password_reset_done),
    url(r'^accounts/password/reset/(?P[0-9A-Za-z]+)-(?P.+)/?$',
        django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '/accounts/password/done/'}),
    url(r'^accounts/password/done/?$',
        django.contrib.auth.views.password_reset_complete),

Extending the User model

This applies to Django 1.4. and just follows the documentation. Add this to settings.py:

AUTH_PROFILE_MODULE = 'pollapp.UserProfile'

and this to pollapp/models.py (changing the specifics to your needs! I’ve said here that users belong to a specific “circle”, and have a flag for whether they have read the terms of service):

from django.contrib.auth.models import User

class UserProfile(models.Model):
    # This field is required.
    user = models.OneToOneField(User)
    circle = models.ForeignKey(Circle)
    read_terms = models.BooleanField()

    def __unicode__(self):
        return self.user.email

from django.db.models.signals import post_save

def create_user_profile(sender, instance, created, **kwargs):
    if created:
        UserProfile.objects.create(user=instance)

post_save.connect(create_user_profile, sender=User)

# This next piece is mine - it changes the appearance of the 
# user in the admin panel (useful if you use an email address
# instead of a username, as we will do in the next section)

def user_display(self):
 return "%s %s (%s)" % (self.first_name, self.last_name, self.email)

User.__unicode__ = user_display

And create pollapp/admin.py:

#
# to extend the user model in the admin, as per
# https://docs.djangoproject.com/en/dev/topics/auth/
#
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth.models import User

# Define an inline admin descriptor for UserProfile model
# which acts a bit like a singleton
class UserProfileInline(admin.StackedInline):
    model = models.UserProfile
    can_delete = False
    verbose_name_plural = 'profile'

# Define a new User admin
class UserAdmin(UserAdmin):
    inlines = (UserProfileInline, )

# Re-register UserAdmin
admin.site.unregister(User)
admin.site.register(User, UserAdmin)

While we’re on this subject, we might as well define the function SetJoinerProfile alluded to before (I’ve put this in its own pollapp/profile.py file, hence the earlier from pollapp.profile import SetJoinerProfile):

from circleapp import models

def SetJoinerProfile(new_user, form):
    """
    An extra function required by RTloginapp,
    to set additional user profile data on first joining.
    The function is provided with the new user object and the JoinForm.
    It should finish by saving the user profile.
    """
    new_user_profile = new_user.get_profile()
    circle = form.cleaned_data["circle"]
    new_user_profile.cricle = circle
    new_user_profile.read_terms = form.cleaned_data['read_terms']
    new_user_profile.save()  # is this the standard approach?
    return

Note the last line is to save the new user’s profile. I haven’t found any reference online to needing to do this – I would have thought saving the user would automatically save the profile too (the user is saved by RTloginapp immediately after calling the above function). But I find that it’s not saved if you don’t explicitly save the profile.

Using an email address instead of a username

I personally like using the email address instead of a username – it’s one less thing to have to remember. But Django 1.4 needs a unique username (max of 30 chars, so I can’t really use the email address as the username). My solution is simply to hash the email address and use the first 30 characters as the username. I include a check for hash collisions, to be on the safe side, though I’m sure it’s overkill.

To make this easier, I’ve added a “backend” so that you can look up a user based on their email address, following this post.  This will allow us later to authenticate users with the command:

   user = authenticate(username=email, password=password)

For this you need to add to settings.py:

AUTHENTICATION_BACKENDS = (
    'RTloginapp.accounts.backends.EmailUsernameBackend',
    'django.contrib.auth.backends.ModelBackend'
)

I then added an accounts directory under RTloginapp, with a blank __init__.py file, and a file backends.py which contains:

from django.conf import settings
from django.contrib.auth.models import User
from RTloginapp import utils

class EmailUsernameBackend(object):
    def authenticate(self, username=None, password=None):
        """
        If the username is an email address,
        then get the user with that email address.
        Otherwise get the user with that username.
        """
        if '@' in username:
            (uname, user) = utils.get_username_and_user(email=username)
            # if that couldn't find the user, it's possible the user's
            # username was entered manually, so just look up the email address
            if user is None:
                user = utils.get_object_or_none(User, email=username)
        else:
            user = utils.get_object_or_none(User, username=username)
        if user is not None:
            if user.check_password(password):
                return user
        return None

    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except User.DoesNotExist:
            return None

This uses some utility functions I put together and saved as RTloginapp/utils.py:

def get_object_or_none(model, **kwargs):
    """
    Sample usage: user = get_or_none(User, username='me')
    """
    try:
        return model.objects.get(**kwargs)
    except model.DoesNotExist:
        return None

from django.contrib.auth.models import User
import hashlib

def _get_user(username, email):
    """Used by get_username_and_user"""
    user = get_object_or_none(User, username=username)
    if user is not None:
        if user.email != email:
            # remotely possible the hashes collide
            username = 'x'+username[:29]
            (username, user) = _get_user(username, email)
    return (username, user)

def get_username_and_user(email):
    """
    Pass the user's email address, will return a tuple of the username
    (almost always the hash of the email)
    and the user object if it exists
    """
    username = hashlib.md5(email).hexdigest()[:30]
    return _get_user(username, email)

Outstanding – how do I get the user’s identification in the admin console to show as the email address, instead of the now unintelligible username?

Forms

This is a good place to introduce the JoinForm. Let’s define the minimal form required by the login app, by putting the following into RTloginapp/forms.py. Note the use of widgets to insert placeholder text in the form.

from django import forms

class BaseRTJoinForm(forms.Form):
    name = forms.CharField(max_length=100, label="Your name",
        error_messages={'required': 'Please enter your name.'},
        widget=forms.TextInput(attrs={'placeholder': 'Your name'}))
    email = forms.EmailField(label="Email address",
        error_messages={'required': 'Please enter your email address.'},
        widget=forms.TextInput(attrs={'placeholder': 'Email address'}))
    password1 = forms.CharField(label="Choose a password",
        error_messages={'required': 'Please choose a password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Choose a password'}))
    password2 = forms.CharField(label="Re-type password",
        error_messages={'required': 'Please re-type your password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Re-type password'}))

    def clean(self):
        cleaned_data = super(BaseRTJoinForm, self).clean()
        password1 = cleaned_data.get("password1")
        password2 = cleaned_data.get("password2")

        if password1 and password2 and password1!=password2:
            msg = r"Your re-typed password does not match the first one. Please type them both again!"
            self._errors["password1"] = self.error_class([msg])
            del cleaned_data["password1"]
            del cleaned_data["password2"]

        # Always return the full collection of cleaned data.
        return cleaned_data

This just follows the principles outlined in the Django documentation.

Then add the customised join form in pollapp/forms.py, eg.:

from django import forms
from RTloginapp.forms import BaseRTJoinForm
from pollapp import models

class JoinForm(BaseRTJoinForm):
    circle = forms.ModelChoiceField(queryset=models.Circle.objects.all(),
        label="Your circle",
        error_messages={'required': 'Which circle are you in?'},
        widget=forms.Select(attrs={'data-theme': 'c', 'data-inline':'true'}))
    read_terms = forms.BooleanField(label="I agree",
        error_messages={'required': 'Please agree to the terms to join.'},
        widget=forms.CheckboxInput(attrs={'data-theme': 'c', 'data-inline':'true'}))

Sending email

We also need to be able to send emails.  This is covered nicely in this stackoverflow post. Add this to utils.py:

def send_email(email_template_name, to_emails, context_dict={}):
    """
    Sends an email based on a template (e.g. login/email/join.html).
    Requires the following files:
    email_template_name.html - html email content, rendered as a template
    email_template_name.txt - plain text content of the email as a template
    email_template_name-subject.txt - a single line with the subject as a template
    email_template_name-from.txt  - a single email address stating who it is from
    to_emails can either be a single email in a string, or a list.
    """
    from django.core.mail import EmailMultiAlternatives
    from django.template.loader import get_template
    from django.template import Context

    if isinstance(to_emails, basestring):
        to_emails = [to_emails]

    subjecttext = get_template(email_template_name+'-subject.txt')
    fromtext = get_template(email_template_name+'-from.txt')
    plaintext = get_template(email_template_name+'.txt')
    htmly     = get_template(email_template_name+'.html')
    context = Context(context_dict)

    text_from    = fromtext.render(context)
    text_subject = subjecttext.render(context)
    text_content = plaintext.render(context)
    html_content = htmly.render(context)

    msg = EmailMultiAlternatives(text_subject, text_content, text_from, to_emails)
    msg.attach_alternative(html_content, "text/html")
    if text_subject and text_from and msg and to_emails:
        try:
            msg.send()
        except:
            return False
        else:
            return True
    return False

However, to actually send emails, you also need to add to settings.py (changing the host user and password to match a gmail account that you own) as explained in this stackoverflow post:

    EMAIL_USE_TLS = True
    EMAIL_HOST = 'smtp.gmail.com'
    EMAIL_HOST_USER = '@gmail.com'
    EMAIL_HOST_PASSWORD = 'example-password'
    EMAIL_PORT = 587

Outstanding – PHP lets me send an email from any user. Django only lets me send it from the host user above (it seems to ignore the “from” field in the send() call above). Is there a way to send from different email addresses in Django depending on the function, and even ones that are not real, like “do-not-reply@…”  ?

Sites

The password reset process uses Sites.

If you haven’t already, you’ll need to add your sites using python manage.py shell, along the lines of:

   
>>> from django.contrib.sites.models import Site
>>> s = Site.objects.get(pk=1)
>>> s.domain = 'mysite.com'
>>> s.name = 'My Site'
>>> s.save()
>>> s2 = Site(domain='localhost:8000', name='My local host')
>>> s2.save()
>>> s2.id

In my settings.py I also test if DEBUG is True, and set SITE_ID to 2 if so.

The views

That covers all the set up. Now we need to define the behaviour of four pages:

  • Sign in
  • Join (including sending the new user an email so they can activate their account)
  • Activate a new user when they follow the email link
  • Sign out

Let’s go through them in order:

Sign in

from django.shortcuts import get_object_or_404, render, render_to_response
from django.http import HttpResponseRedirect, HttpResponse
from django.contrib.auth import authenticate, login, logout

def signin(request, return_url='/', **kwargs):
    context = kwargs
    context['return_url'] = return_url
    email = request.POST.get('email', '')
    if email=='':
        # first time here (or no email), so display login form
        return render(request, "login/login.html", context)
    else:
        # is it a valid user? note this utilises the emailUsernameBackend
        password = request.POST.get('password', '')
        user = authenticate(username=email, password=password)
        if user is not None and user.is_active:
            # Correct password, and the user is marked "active"
            login(request, user)
            stay = request.POST.get('stay', False)
            if stay:
                request.session.set_expiry(31*24*60*60) # a month if click "stay logged in"
            else:
                request.session.set_expiry(2*60*60) # or else just two hours
            # Redirect to a success page.
            return HttpResponseRedirect(return_url)
        else:
            # Show login form with error message
            context['invalid'] = True
            return render(request, "login/login.html", context)

This clearly depends on a template called login/login.html, which we’ll come to in the next section. I’ll summarise the context for the templates there too.

Outstanding – it would be nice to set the time-limits for logging in in the settings.py file. (Alternatively they could be parameters passed in via urls.py.)

Join

from django import forms
from django.db import IntegrityError

from RTloginapp import models, utils
from RTloginapp.forms import BaseRTJoinForm

def join(request, JoinForm=BaseRTJoinForm, SetJoinerProfile=None, return_url="/", forgot_link="/accounts/password/reset", activate_link="/accounts/register", **kwargs):
    """The user is "active" if they have confirmed their email address by
       responding to the join email.
       The user is "inactive" if they have filled in the join form but not
       yet responded to the email, or if they are removed for some reason.
       An inactive user cannot sign in.
       If a user tries to join using the email address of an inactive user,
       the old user is updated, their name and related info is overwritten,
       and an email is sent out so they can become active.
    """
    context = kwargs
    context['return_url'] = return_url
    if request.method == 'POST':
        form = JoinForm(request.POST)
        context['form'] = form
        if form.is_valid():
            (username, user) = utils.get_username_and_user(form.cleaned_data['email'])
            if user is None:
                new_user = User.objects.create_user(
                                username=username,
                                password=form.cleaned_data['password1'],
                                email=form.cleaned_data['email'])
            elif user.is_active:
                # if the user has an active account,
                # send an email with a link to "forgot password" instead
                context['new_user'] = user
                context['forgot_link'] = request.build_absolute_uri(forgot_link)
                context['email_ok'] = utils.send_email("login/email/already-joined", form.cleaned_data['email'], context_dict=context)
                return render(request, "login/join-email-sent.html", context)
            else:
                # the user is inactive, so update the user instead
                new_user = user
            #
            new_user.first_name = ' '.join(form.cleaned_data['name'].split(' ')[:-1])
            new_user.last_name = form.cleaned_data['name'].split(' ')[-1]
            new_user.is_active = False
            if SetJoinerProfile is not None:
                SetJoinerProfile(new_user, form)
            new_user.save()
            #
            context['new_user'] = new_user
            context['reg_link'] = request.build_absolute_uri(activate_link + "?l="+username)
            context['email_ok'] = utils.send_email("login/email/join", new_user.email, context_dict=context)
            return render(request, "login/join-email-sent.html", context)
    else:
        form = JoinForm()
        context['form'] = form
    return render(request, "login/join.html", context)

How does that look? Please let me know if you see any security problems with it. Note I also split out the first and last names from a single name field.  This split is not perfect, so I plan to always just display the two together.

Activate

A link to this page is emailed to the user when they join, with the username appended as a GET parameter. All the page does is check the GET parameter is valid, and activate the user’s account if so. The emailed link remains valid indefinitely, which may be not be ideal.

def activate(request):
    if request.method == 'GET':
        reg_id = request.GET.get('l', '')
        if reg_id=='':
            return HttpResponseRedirect("/accounts/join")
        else:
            user = utils.get_object_or_none(User, username=reg_id)
            if user is not None:
                if not user.is_active:
                    user.is_active=True
                    user.save()
                    return render(request, "login/activated.html", {'new_user': user})
                else:
                    return render(request, "login/already-active.html", {'new_user': user})
    return HttpResponseRedirect("/accounts/join")

Signout

def signout(request, return_url="/"):
    logout(request)
    return HttpResponseRedirect(return_url)

The templates

Set up a series of website and email templates, which will have the context shown at right:

   login/
        login.html              invalid (only if True)
        join-email-sent.html    form, new_user, email_ok  (and forgot_link or reg_link)
        join.html               form
        already-active.html     new_user (*)
        activated.html          new_user (*)

    login/email/
        already-joined          form, new_user, forgot_link
        join                    form, new_user, reg_link

Each of the two email templates requires four files: .html and .txt for the content, and -from.txt and -subject.txt (these should only be a single line).

For each of the above except *, the context also includes:

                               return_url
                                any keyword args passed in by the url scheme

You will also need the reset-password templates – there are some samples available in the Django documentation:

   registration/
        password_reset_complete.html
        password_reset_confirm.html
        password_reset_done.html
        password_reset_email.html
        password_reset_form.html

This post is already ridiculously long, so I’ll just show a few of the templates I’m using. I put these in pollapp/templates/, and added this directory to settings.py‘s TEMPLATE_DIRS. The details will depend on your base template and what framework you’re using; this is for jQuery Mobile. It is a little painful to have to create all these files.

login/join.html

{% extends "base.html" %}
{% block title %}Join{% endblock %}

{% block content %}</pre>
<form action="" method="post" data-ajax="false">
{% for error in form.non_field_errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% csrf_token %}
 {% for hidden in form.hidden_fields %} {{ hidden }} {% endfor %}
 {% for field in form.visible_fields %}
 {% for error in field.errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% ifequal field.auto_id "id_read_terms" %}
 Finally... please read and agree to our
 <a href="/terms" target="readterms" data-ajax="false">terms of service</a>
 {{ field }}
 <label for="{{ field.auto_id }}">{{ field.label }}</label>
 {% else %}
 <label for="{{ field.auto_id }}">{{ field.label }}:</label>
 {{ field }}
 {% endifequal %}
 {% endfor %}
<div class="right"><a href="{{ return_url }}" data-role="button" data-theme="c" data-inline="true" data-ajax="false">Cancel </a>
 <input type="submit" name="joinBtn" value="Join" data-theme="b" data-inline="true" /></div>
</form>
<pre>{% endblock %}

This is a good place to point out a strange thing about templates – I had to use “ifequal“, not “if a==b“; the latter produced an error.

login/signin.html

{% extends "base.html" %}
{% block title %}Sign in{% endblock %}
{% block content %}
  {% if form.errors or invalid %}
  Sorry, that's not a valid username or password
  {% endif %}

  Please sign in</pre>
<form action="/accounts/login/" method="POST" data-ajax="false">
{% csrf_token %}
<fieldset data-role="fieldcontain"><label class="ui-hidden-accessible" for="email">Email</label>
 <input id="email" type="email" name="email" placeholder="Email" size="35/" data-theme="b" />
 <label class="ui-hidden-accessible" for="password">Password</label>
 <span class="left"><input id="password" type="password" name="password" placeholder="Password" data-theme="b" /></span>
 <span class="right"><input type="submit" name="login" value="OK" data-theme="b" data-inline="true" /></span></fieldset>
<fieldset><span class="left">
 <input id="stay" type="checkbox" name="stay" data-theme="a" data-inline="true" data-mini="true" />
 <label for="stay">Stay signed in</label>
 </span>
 <span class="right">
 <a href="/accounts/password/reset" data-ajax="false">Forgot password?</a>
 </span></fieldset>
</form>
<pre>{% endblock %}

login/email/join.html

Thank you for asking to join my site.
To complete your request, please click the following link:
<a href="{{ reg_link }}">{{ reg_link }}</a>
We ask this to make sure that your email address has not been used by someone else.
If you did not ask to join, please ignore this email, or
email our administrators.
Thank you
My Site

Conclusion

If you followed the above, and added a few more templates of your own, you should have a reasonably functioning user registration system. (I make no guarantees though – use it at your own risk!) In fact, if that’s what you want, I would suggest you take a look at django-registration instead, although I think that still requires a username in addition to an email address.

My real aim in writing this up, though, is to answer some of the newbie questions I had about Django and to help you avoid some of the trouble spots. In particular, how to do some very basic things, like:

  • decide what’s an app and what’s a project?
  • use an email address instead of a username for user registration?
  • extend the user model?
  • send (nicely formatted) emails?
  • build forms so that extra validation can be automatically included?
  • include extra fields like “placeholder” in forms?

I hope I’ve helped to answer all these questions in this post.  Let me know!

Also, look out for my next post on Django, which will cover my discoveries using extensions like Django-CMS, South and virtualenv.