All posts by Arthur Street

Django with Django-CMS

I recently found myself starting to design a flexible model for the different pages of my Django website.  After an enjoyable morning thinking through how to structure the data it hit me: I was designing a content management system! Surely someone has done that before… So I turned my attention to Django-CMS.

To get a basic Django-CMS project running there is an impressive list of changes you need to make to your new project’s settings, but if you follow the tutorial it all works well, and you can get a page template of your own design up and running so that someone else can add content to it through Django’s very nice admin console.  Note – make sure you follow the tutorial for the same version of Django-CMS as you have installed!  Somehow I wound up following an older tutorial and it has caused me a lot of pain – lots of settings were subtly different.

Adding the tables

I found myself going around in circles trying to get South to put the Django-CMS tables into my pre-existing project, until I found this stackoverflow post.  The accepted answer is commenting out south from the INSTALLED_APPS, then running the usual python manage.py syncdb command, then uncommenting it and then doing a fake migration (manage.py migrate --fake).  Stuff like this makes me wonder if I’m really making my life any easier using Django… perhaps I should chuck it in and use one of the more popular frameworks? (But then I also remember how hard it was to move this blog off the front page of my WordPress site…)

Adding your own css and js

I see that Django-CMS uses some other packages, including django-sekizai to render css and javascript. I want to put my own css file in of course, and use Twitter Bootstrap too. From the django-sekizai documentation, you just need to add:

{% addtoblock "css" %}
	<link href="/media/css/stylesheet.css" rel="stylesheet" media="screen" type="text/css">
{% endaddtoblock %}

along with something similar for js.

However, there is a trick here, elucidated in this stackoverflow post.  These static files that you add must go into a differently named folder like local-static in your app directory, and you must add this to your settings.py:

STATICFILES_DIRS = (
    # Put strings here, like /home/html/static or C:/www/django/static.
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(PROJECT_PATH, "local-static"),
)

Talk about a trap for young players!

Outstanding – why is the static file directory inside the app directory after doing this, when the PROJECT_PATH appears to be one level up?

How to integrate an existing app into Django-CMS

This has been a very painful process, and I have yet to complete it. I am using the “AppHook” approach, though the documentation lists this as third out of five methods with little guidance to help you choose.

I made the following changes to my RTloginapp (which is described in this post):

Add a new file RTloginapp/cms_app.py:

from cms.app_base import CMSApp
from cms.apphook_pool import apphook_pool
from django.utils.translation import ugettext_lazy as _

class RTLoginApphook(CMSApp):
    name = _("RTLoginApphook")
    urls = ["RTloginapp.urls"]

apphook_pool.register(RTLoginApphook)

Add a new file RTloginapp/urls.py:

from django.conf.urls.defaults import patterns, include, url
import django.contrib.auth.views

urlpatterns = patterns('RTloginapp.views',

    url(r'^/?$', 'signin', name="signin"),
    url(r'^login/?$', 'signin'),
    url(r'^join/?$', 'join', {"forgot_link":"//password/reset", "activate_link":"//register"}),
    url(r'^register/?$', 'activate'),
    url(r'^signout/?$', 'signout'),

    url(r'^password/reset/?$', django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '//password/reset/done/'}),
    url(r'^password/reset/done/?$', django.contrib.auth.views.password_reset_done),
    url(r'^password/reset/(?P<uidb36>[0-9A-Za-z]+)-(?P<token>.+)/?$', django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '//password/done/'}),
    url(r'^password/done/?$', django.contrib.auth.views.password_reset_complete),
)

And then go into the admin panel and add an “accounts” page with the application (under advanced settings) set to RTLoginApphook.

Note if you muck up the urls reference, I found I needed to uninstall the apphook using:

python manage.py cms uninstall apphooks RTLoginApphook

and then go back to the admin page and hook it back up again to a page.

Note the RTloginapp templates may need to be modified if you have renamed any of the blocks, e.g. the Django-CMS tutorial uses “base_content” as the block name, where I wrote login/login.html to use “content”.

What are menus?

It took me a while to find a description of what was meant by a menu.  I think this Django-CMS documentation page gives a good enough introduction.

My “en” pages are missing!

This turned out to be due to the order of the MIDDLEWARE_CLASSES (a relic of the fact that I had originally followed an older tutorial and fixed it manually by adding cms.middleware.multilingual.MultilingualURLMiddleware to the end… it should be before the other cms.middleware classes). Not an easy error to find.

Summary

I wavered several times on whether to use Django-CMS.  It felt like many of the concepts were too heavy (what are app hooks?), there was limited documentation, and googling only sometimes solved my problems.  But in the end, I was able to push through and start using it, and I have found it very powerful.

  

Getting started with Django

For the past couple of weeks I have been developing a website using Django, a python-based framework for content management. This is after writing the site for some weeks directly in PHP using SQL, without a particular framework. At first the site started out quite simple, so that seemed sensible. As it grew I realised it would someday reach a point of unmanageability, so I have turned to a framework.

Frameworks have several advantages, including cleanly separating the underlying business logic (the model) from the way it is presented (the view); abstracting away from the SQL, instead presenting the objects represented by the SQL; making forms and form validation easier; making user registration easier; making database administration easier; making the URL scheme easier; and having lots of add-ons that make extended functionality easier.  I chose Django because I am a fan of Python.

Django has some great features which make it easy to convert an existing SQL database structure over, with hardly any changes required to the tables. The one sticking point for me has been the user registration. I hesitated for some time before tackling this, because uncharacteristically, the user model in Django 1.4 is fixed, and to extend it requires an additional one-to-one table.  Adding to the uncertainty, whatever you do now will not be supported in Django 1.5, when a more flexible user model will be released.  However, it is fairly easy to implement the extension, and I hope it will be easy to convert to Django 1.5 when the time comes.

So here I want to point out the stumbling blocks that I hit as a total beginner to Django, so that you can help see ahead of time what is involved in learning and using it. My comments come with the caveat that I have been using Django for a matter of weeks, so do not take me as any sort of authority on best practice. I would love feedback on how to do things better!

I think the best way to go is to describe (in a mini-tutorial) how I’ve built my two-phase user registration process, which uses email addresses in place of usernames. You’ll see how I implemented “Sign in” and “Join” pages, how to check a new user’s email is not already in the database, to send an email to the user’s email address when they first join, to only activate the user when they follow a link in that email, and to handle “forgot password” and “stay signed in”.  I’ve done this for a site using jQuery Mobile. Let’s begin!

App or project?

Begin with a project and an app, just like the poll app in the Django tutorial. I originally started building my user registration process as part of this app (I’ll call it pollapp), but have since realised I can separate it out as its own app, which I am calling RTloginapp.  RTloginapp can then be used for other projects where the User model may be different. (Remember to add 'RTloginapp' to the INSTALLED_APPS in settings.py.)

The points where RTloginapp and pollapp need to communicate are:

  • RTloginapp needs customised views, eg. to display the forgot password form, the user signin form etc.  Solution: All the templates are actually put in pollapp. If I was properly packaging up RTloginapp as well, I would put default templates in RTloginapp too. I’d like to have those default templates simply extend a base template so they have more chance of being actually useful… but I’m not sure what the standard practice is here – please let me know if you have an opinion.
  • In the URLs. I have put the RTloginapp URL scheme into pollapp/urls.py, not RTloginapp/urls.py.  This way the url scheme is not fixed for every project, and you can pass the views extra project-specific parameters (the next few points).
  • RTloginapp needs to display a customised first-time user join form, since different projects may need different information about the user. Solution: in urls.py, pass the join view an extra parameter that specifies the join form.
  • RTloginapp needs to save the first-time user’s information when they join. Solution: in urls.py, pass a function as a parameter to the join view. The function saves any extra information.
  • RTloginapp needs to know a few links – the “forgot password” URL and the “activate” URL. These are passed in via urls.py too. You can also optionally pass in a “return” URL which is passed to the templates if required, and any number of additional context settings.

So pollapp/urls.py will look like this when we’re done. Note I’ve added an extra question mark after the final slash in each URL (before the $), so that the final slash in the URL is optional. The official documentation seems to assume URLs will always finish with a slash.

from pollapp.forms import JoinForm
from pollapp.profile import SetJoinerProfile

urlpatterns = patterns('RTloginapp.views',

    url(r'^accounts/?$', 'signin'),
    url(r'^accounts/login/?$', 'signin'),
    url(r'^accounts/join/?$', 'join', {'JoinForm': JoinForm,
        'SetJoinerProfile': SetJoinerProfile,
        "forgot_link":"/accounts/password/reset",
        "activate_link":"/accounts/register"}),
    url(r'^accounts/register/?$', 'activate'),
    url(r'^accounts/signout/?$', 'signout'),
)

urlpatterns += patterns('your-app.views',
		# other urls
)

Django comes with a password-reset facility, which we can use by adding the following import:

    import django.contrib.auth.views

and these URLs:

    url(r'^accounts/password/reset/?$',
        django.contrib.auth.views.password_reset,
        {'post_reset_redirect' : '/accounts/password/reset/done/'}),
    url(r'^accounts/password/reset/done/?$',
        django.contrib.auth.views.password_reset_done),
    url(r'^accounts/password/reset/(?P[0-9A-Za-z]+)-(?P.+)/?$',
        django.contrib.auth.views.password_reset_confirm,
        {'post_reset_redirect' : '/accounts/password/done/'}),
    url(r'^accounts/password/done/?$',
        django.contrib.auth.views.password_reset_complete),

Extending the User model

This applies to Django 1.4. and just follows the documentation. Add this to settings.py:

AUTH_PROFILE_MODULE = 'pollapp.UserProfile'

and this to pollapp/models.py (changing the specifics to your needs! I’ve said here that users belong to a specific “circle”, and have a flag for whether they have read the terms of service):

from django.contrib.auth.models import User

class UserProfile(models.Model):
    # This field is required.
    user = models.OneToOneField(User)
    circle = models.ForeignKey(Circle)
    read_terms = models.BooleanField()

    def __unicode__(self):
        return self.user.email

from django.db.models.signals import post_save

def create_user_profile(sender, instance, created, **kwargs):
    if created:
        UserProfile.objects.create(user=instance)

post_save.connect(create_user_profile, sender=User)

# This next piece is mine - it changes the appearance of the 
# user in the admin panel (useful if you use an email address
# instead of a username, as we will do in the next section)

def user_display(self):
 return "%s %s (%s)" % (self.first_name, self.last_name, self.email)

User.__unicode__ = user_display

And create pollapp/admin.py:

#
# to extend the user model in the admin, as per
# https://docs.djangoproject.com/en/dev/topics/auth/
#
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth.models import User

# Define an inline admin descriptor for UserProfile model
# which acts a bit like a singleton
class UserProfileInline(admin.StackedInline):
    model = models.UserProfile
    can_delete = False
    verbose_name_plural = 'profile'

# Define a new User admin
class UserAdmin(UserAdmin):
    inlines = (UserProfileInline, )

# Re-register UserAdmin
admin.site.unregister(User)
admin.site.register(User, UserAdmin)

While we’re on this subject, we might as well define the function SetJoinerProfile alluded to before (I’ve put this in its own pollapp/profile.py file, hence the earlier from pollapp.profile import SetJoinerProfile):

from circleapp import models

def SetJoinerProfile(new_user, form):
    """
    An extra function required by RTloginapp,
    to set additional user profile data on first joining.
    The function is provided with the new user object and the JoinForm.
    It should finish by saving the user profile.
    """
    new_user_profile = new_user.get_profile()
    circle = form.cleaned_data["circle"]
    new_user_profile.cricle = circle
    new_user_profile.read_terms = form.cleaned_data['read_terms']
    new_user_profile.save()  # is this the standard approach?
    return

Note the last line is to save the new user’s profile. I haven’t found any reference online to needing to do this – I would have thought saving the user would automatically save the profile too (the user is saved by RTloginapp immediately after calling the above function). But I find that it’s not saved if you don’t explicitly save the profile.

Using an email address instead of a username

I personally like using the email address instead of a username – it’s one less thing to have to remember. But Django 1.4 needs a unique username (max of 30 chars, so I can’t really use the email address as the username). My solution is simply to hash the email address and use the first 30 characters as the username. I include a check for hash collisions, to be on the safe side, though I’m sure it’s overkill.

To make this easier, I’ve added a “backend” so that you can look up a user based on their email address, following this post.  This will allow us later to authenticate users with the command:

   user = authenticate(username=email, password=password)

For this you need to add to settings.py:

AUTHENTICATION_BACKENDS = (
    'RTloginapp.accounts.backends.EmailUsernameBackend',
    'django.contrib.auth.backends.ModelBackend'
)

I then added an accounts directory under RTloginapp, with a blank __init__.py file, and a file backends.py which contains:

from django.conf import settings
from django.contrib.auth.models import User
from RTloginapp import utils

class EmailUsernameBackend(object):
    def authenticate(self, username=None, password=None):
        """
        If the username is an email address,
        then get the user with that email address.
        Otherwise get the user with that username.
        """
        if '@' in username:
            (uname, user) = utils.get_username_and_user(email=username)
            # if that couldn't find the user, it's possible the user's
            # username was entered manually, so just look up the email address
            if user is None:
                user = utils.get_object_or_none(User, email=username)
        else:
            user = utils.get_object_or_none(User, username=username)
        if user is not None:
            if user.check_password(password):
                return user
        return None

    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except User.DoesNotExist:
            return None

This uses some utility functions I put together and saved as RTloginapp/utils.py:

def get_object_or_none(model, **kwargs):
    """
    Sample usage: user = get_or_none(User, username='me')
    """
    try:
        return model.objects.get(**kwargs)
    except model.DoesNotExist:
        return None

from django.contrib.auth.models import User
import hashlib

def _get_user(username, email):
    """Used by get_username_and_user"""
    user = get_object_or_none(User, username=username)
    if user is not None:
        if user.email != email:
            # remotely possible the hashes collide
            username = 'x'+username[:29]
            (username, user) = _get_user(username, email)
    return (username, user)

def get_username_and_user(email):
    """
    Pass the user's email address, will return a tuple of the username
    (almost always the hash of the email)
    and the user object if it exists
    """
    username = hashlib.md5(email).hexdigest()[:30]
    return _get_user(username, email)

Outstanding – how do I get the user’s identification in the admin console to show as the email address, instead of the now unintelligible username?

Forms

This is a good place to introduce the JoinForm. Let’s define the minimal form required by the login app, by putting the following into RTloginapp/forms.py. Note the use of widgets to insert placeholder text in the form.

from django import forms

class BaseRTJoinForm(forms.Form):
    name = forms.CharField(max_length=100, label="Your name",
        error_messages={'required': 'Please enter your name.'},
        widget=forms.TextInput(attrs={'placeholder': 'Your name'}))
    email = forms.EmailField(label="Email address",
        error_messages={'required': 'Please enter your email address.'},
        widget=forms.TextInput(attrs={'placeholder': 'Email address'}))
    password1 = forms.CharField(label="Choose a password",
        error_messages={'required': 'Please choose a password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Choose a password'}))
    password2 = forms.CharField(label="Re-type password",
        error_messages={'required': 'Please re-type your password.'},
        widget=forms.PasswordInput(render_value=True,
            attrs={'placeholder': 'Re-type password'}))

    def clean(self):
        cleaned_data = super(BaseRTJoinForm, self).clean()
        password1 = cleaned_data.get("password1")
        password2 = cleaned_data.get("password2")

        if password1 and password2 and password1!=password2:
            msg = r"Your re-typed password does not match the first one. Please type them both again!"
            self._errors["password1"] = self.error_class([msg])
            del cleaned_data["password1"]
            del cleaned_data["password2"]

        # Always return the full collection of cleaned data.
        return cleaned_data

This just follows the principles outlined in the Django documentation.

Then add the customised join form in pollapp/forms.py, eg.:

from django import forms
from RTloginapp.forms import BaseRTJoinForm
from pollapp import models

class JoinForm(BaseRTJoinForm):
    circle = forms.ModelChoiceField(queryset=models.Circle.objects.all(),
        label="Your circle",
        error_messages={'required': 'Which circle are you in?'},
        widget=forms.Select(attrs={'data-theme': 'c', 'data-inline':'true'}))
    read_terms = forms.BooleanField(label="I agree",
        error_messages={'required': 'Please agree to the terms to join.'},
        widget=forms.CheckboxInput(attrs={'data-theme': 'c', 'data-inline':'true'}))

Sending email

We also need to be able to send emails.  This is covered nicely in this stackoverflow post. Add this to utils.py:

def send_email(email_template_name, to_emails, context_dict={}):
    """
    Sends an email based on a template (e.g. login/email/join.html).
    Requires the following files:
    email_template_name.html - html email content, rendered as a template
    email_template_name.txt - plain text content of the email as a template
    email_template_name-subject.txt - a single line with the subject as a template
    email_template_name-from.txt  - a single email address stating who it is from
    to_emails can either be a single email in a string, or a list.
    """
    from django.core.mail import EmailMultiAlternatives
    from django.template.loader import get_template
    from django.template import Context

    if isinstance(to_emails, basestring):
        to_emails = [to_emails]

    subjecttext = get_template(email_template_name+'-subject.txt')
    fromtext = get_template(email_template_name+'-from.txt')
    plaintext = get_template(email_template_name+'.txt')
    htmly     = get_template(email_template_name+'.html')
    context = Context(context_dict)

    text_from    = fromtext.render(context)
    text_subject = subjecttext.render(context)
    text_content = plaintext.render(context)
    html_content = htmly.render(context)

    msg = EmailMultiAlternatives(text_subject, text_content, text_from, to_emails)
    msg.attach_alternative(html_content, "text/html")
    if text_subject and text_from and msg and to_emails:
        try:
            msg.send()
        except:
            return False
        else:
            return True
    return False

However, to actually send emails, you also need to add to settings.py (changing the host user and password to match a gmail account that you own) as explained in this stackoverflow post:

    EMAIL_USE_TLS = True
    EMAIL_HOST = 'smtp.gmail.com'
    EMAIL_HOST_USER = '@gmail.com'
    EMAIL_HOST_PASSWORD = 'example-password'
    EMAIL_PORT = 587

Outstanding – PHP lets me send an email from any user. Django only lets me send it from the host user above (it seems to ignore the “from” field in the send() call above). Is there a way to send from different email addresses in Django depending on the function, and even ones that are not real, like “do-not-reply@…”  ?

Sites

The password reset process uses Sites.

If you haven’t already, you’ll need to add your sites using python manage.py shell, along the lines of:

   
>>> from django.contrib.sites.models import Site
>>> s = Site.objects.get(pk=1)
>>> s.domain = 'mysite.com'
>>> s.name = 'My Site'
>>> s.save()
>>> s2 = Site(domain='localhost:8000', name='My local host')
>>> s2.save()
>>> s2.id

In my settings.py I also test if DEBUG is True, and set SITE_ID to 2 if so.

The views

That covers all the set up. Now we need to define the behaviour of four pages:

  • Sign in
  • Join (including sending the new user an email so they can activate their account)
  • Activate a new user when they follow the email link
  • Sign out

Let’s go through them in order:

Sign in

from django.shortcuts import get_object_or_404, render, render_to_response
from django.http import HttpResponseRedirect, HttpResponse
from django.contrib.auth import authenticate, login, logout

def signin(request, return_url='/', **kwargs):
    context = kwargs
    context['return_url'] = return_url
    email = request.POST.get('email', '')
    if email=='':
        # first time here (or no email), so display login form
        return render(request, "login/login.html", context)
    else:
        # is it a valid user? note this utilises the emailUsernameBackend
        password = request.POST.get('password', '')
        user = authenticate(username=email, password=password)
        if user is not None and user.is_active:
            # Correct password, and the user is marked "active"
            login(request, user)
            stay = request.POST.get('stay', False)
            if stay:
                request.session.set_expiry(31*24*60*60) # a month if click "stay logged in"
            else:
                request.session.set_expiry(2*60*60) # or else just two hours
            # Redirect to a success page.
            return HttpResponseRedirect(return_url)
        else:
            # Show login form with error message
            context['invalid'] = True
            return render(request, "login/login.html", context)

This clearly depends on a template called login/login.html, which we’ll come to in the next section. I’ll summarise the context for the templates there too.

Outstanding – it would be nice to set the time-limits for logging in in the settings.py file. (Alternatively they could be parameters passed in via urls.py.)

Join

from django import forms
from django.db import IntegrityError

from RTloginapp import models, utils
from RTloginapp.forms import BaseRTJoinForm

def join(request, JoinForm=BaseRTJoinForm, SetJoinerProfile=None, return_url="/", forgot_link="/accounts/password/reset", activate_link="/accounts/register", **kwargs):
    """The user is "active" if they have confirmed their email address by
       responding to the join email.
       The user is "inactive" if they have filled in the join form but not
       yet responded to the email, or if they are removed for some reason.
       An inactive user cannot sign in.
       If a user tries to join using the email address of an inactive user,
       the old user is updated, their name and related info is overwritten,
       and an email is sent out so they can become active.
    """
    context = kwargs
    context['return_url'] = return_url
    if request.method == 'POST':
        form = JoinForm(request.POST)
        context['form'] = form
        if form.is_valid():
            (username, user) = utils.get_username_and_user(form.cleaned_data['email'])
            if user is None:
                new_user = User.objects.create_user(
                                username=username,
                                password=form.cleaned_data['password1'],
                                email=form.cleaned_data['email'])
            elif user.is_active:
                # if the user has an active account,
                # send an email with a link to "forgot password" instead
                context['new_user'] = user
                context['forgot_link'] = request.build_absolute_uri(forgot_link)
                context['email_ok'] = utils.send_email("login/email/already-joined", form.cleaned_data['email'], context_dict=context)
                return render(request, "login/join-email-sent.html", context)
            else:
                # the user is inactive, so update the user instead
                new_user = user
            #
            new_user.first_name = ' '.join(form.cleaned_data['name'].split(' ')[:-1])
            new_user.last_name = form.cleaned_data['name'].split(' ')[-1]
            new_user.is_active = False
            if SetJoinerProfile is not None:
                SetJoinerProfile(new_user, form)
            new_user.save()
            #
            context['new_user'] = new_user
            context['reg_link'] = request.build_absolute_uri(activate_link + "?l="+username)
            context['email_ok'] = utils.send_email("login/email/join", new_user.email, context_dict=context)
            return render(request, "login/join-email-sent.html", context)
    else:
        form = JoinForm()
        context['form'] = form
    return render(request, "login/join.html", context)

How does that look? Please let me know if you see any security problems with it. Note I also split out the first and last names from a single name field.  This split is not perfect, so I plan to always just display the two together.

Activate

A link to this page is emailed to the user when they join, with the username appended as a GET parameter. All the page does is check the GET parameter is valid, and activate the user’s account if so. The emailed link remains valid indefinitely, which may be not be ideal.

def activate(request):
    if request.method == 'GET':
        reg_id = request.GET.get('l', '')
        if reg_id=='':
            return HttpResponseRedirect("/accounts/join")
        else:
            user = utils.get_object_or_none(User, username=reg_id)
            if user is not None:
                if not user.is_active:
                    user.is_active=True
                    user.save()
                    return render(request, "login/activated.html", {'new_user': user})
                else:
                    return render(request, "login/already-active.html", {'new_user': user})
    return HttpResponseRedirect("/accounts/join")

Signout

def signout(request, return_url="/"):
    logout(request)
    return HttpResponseRedirect(return_url)

The templates

Set up a series of website and email templates, which will have the context shown at right:

   login/
        login.html              invalid (only if True)
        join-email-sent.html    form, new_user, email_ok  (and forgot_link or reg_link)
        join.html               form
        already-active.html     new_user (*)
        activated.html          new_user (*)

    login/email/
        already-joined          form, new_user, forgot_link
        join                    form, new_user, reg_link

Each of the two email templates requires four files: .html and .txt for the content, and -from.txt and -subject.txt (these should only be a single line).

For each of the above except *, the context also includes:

                               return_url
                                any keyword args passed in by the url scheme

You will also need the reset-password templates – there are some samples available in the Django documentation:

   registration/
        password_reset_complete.html
        password_reset_confirm.html
        password_reset_done.html
        password_reset_email.html
        password_reset_form.html

This post is already ridiculously long, so I’ll just show a few of the templates I’m using. I put these in pollapp/templates/, and added this directory to settings.py‘s TEMPLATE_DIRS. The details will depend on your base template and what framework you’re using; this is for jQuery Mobile. It is a little painful to have to create all these files.

login/join.html

{% extends "base.html" %}
{% block title %}Join{% endblock %}

{% block content %}</pre>
<form action="" method="post" data-ajax="false">
{% for error in form.non_field_errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% csrf_token %}
 {% for hidden in form.hidden_fields %} {{ hidden }} {% endfor %}
 {% for field in form.visible_fields %}
 {% for error in field.errors %}
<div class="warning">{{ error }}</div>
 {% endfor %}
 {% ifequal field.auto_id "id_read_terms" %}
 Finally... please read and agree to our
 <a href="/terms" target="readterms" data-ajax="false">terms of service</a>
 {{ field }}
 <label for="{{ field.auto_id }}">{{ field.label }}</label>
 {% else %}
 <label for="{{ field.auto_id }}">{{ field.label }}:</label>
 {{ field }}
 {% endifequal %}
 {% endfor %}
<div class="right"><a href="{{ return_url }}" data-role="button" data-theme="c" data-inline="true" data-ajax="false">Cancel </a>
 <input type="submit" name="joinBtn" value="Join" data-theme="b" data-inline="true" /></div>
</form>
<pre>{% endblock %}

This is a good place to point out a strange thing about templates – I had to use “ifequal“, not “if a==b“; the latter produced an error.

login/signin.html

{% extends "base.html" %}
{% block title %}Sign in{% endblock %}
{% block content %}
  {% if form.errors or invalid %}
  Sorry, that's not a valid username or password
  {% endif %}

  Please sign in</pre>
<form action="/accounts/login/" method="POST" data-ajax="false">
{% csrf_token %}
<fieldset data-role="fieldcontain"><label class="ui-hidden-accessible" for="email">Email</label>
 <input id="email" type="email" name="email" placeholder="Email" size="35/" data-theme="b" />
 <label class="ui-hidden-accessible" for="password">Password</label>
 <span class="left"><input id="password" type="password" name="password" placeholder="Password" data-theme="b" /></span>
 <span class="right"><input type="submit" name="login" value="OK" data-theme="b" data-inline="true" /></span></fieldset>
<fieldset><span class="left">
 <input id="stay" type="checkbox" name="stay" data-theme="a" data-inline="true" data-mini="true" />
 <label for="stay">Stay signed in</label>
 </span>
 <span class="right">
 <a href="/accounts/password/reset" data-ajax="false">Forgot password?</a>
 </span></fieldset>
</form>
<pre>{% endblock %}

login/email/join.html

Thank you for asking to join my site.
To complete your request, please click the following link:
<a href="{{ reg_link }}">{{ reg_link }}</a>
We ask this to make sure that your email address has not been used by someone else.
If you did not ask to join, please ignore this email, or
email our administrators.
Thank you
My Site

Conclusion

If you followed the above, and added a few more templates of your own, you should have a reasonably functioning user registration system. (I make no guarantees though – use it at your own risk!) In fact, if that’s what you want, I would suggest you take a look at django-registration instead, although I think that still requires a username in addition to an email address.

My real aim in writing this up, though, is to answer some of the newbie questions I had about Django and to help you avoid some of the trouble spots. In particular, how to do some very basic things, like:

  • decide what’s an app and what’s a project?
  • use an email address instead of a username for user registration?
  • extend the user model?
  • send (nicely formatted) emails?
  • build forms so that extra validation can be automatically included?
  • include extra fields like “placeholder” in forms?

I hope I’ve helped to answer all these questions in this post.  Let me know!

Also, look out for my next post on Django, which will cover my discoveries using extensions like Django-CMS, South and virtualenv.

  

Alignment problem with jquerymobile and bold text – only on PC with Chrome

The following html and css works fine on most browsers, but try it on Chrome on a PC:

<!DOCTYPE  HTML>
<html>
<head>
	<link rel="stylesheet" href="http://code.jquery.com/mobile/1.2.0/jquery.mobile-1.2.0.min.css" />
	<script src="http://code.jquery.com/jquery-1.8.2.min.js"></script>
	<script src="http://code.jquery.com/mobile/1.2.0/jquery.mobile-1.2.0.min.js"></script>
</head>

<body>
	<style type="text/css">
		.detail { display: block; }
		.detail .fieldname { font-weight: bold; float: left; }
	</style>
	<div class='detail'>
		<div class='fieldname'>Name:</div><div>Mr Smith</div>
		<div class='fieldname'>Phone:</div><div>xxxx-xxxx</div>
	</div>
</body>
</html>

There is an alignment problem on Chrome on a PC.  Not on Chrome on a Mac, and not on IE on a PC – just Chrome on a PC.  You’ll see the word Phone starts under Mr Smith, instead of under Name.  (I am using the latest version of Chrome as I write this, 23.0.1271.64 m.)

This seems to be a strange interplay between the the bold text and jquery mobile. If you switch off either of these things, it works fine.

In fact there is a simple work-around: change font-weight: bold to font-weight: 600.  It won’t be quite as bold (which is weight 700), but hopefully it will be close enough.

All very mysterious!

  

Lua for scripting NPC behaviour

With character movement looking good, now it’s time to give the non-player-characters (NPCs in role-playing game parlance) some behaviour.

I have the idea that monkeys might chase after bananas, foxes after chickens, etc.  And what platform game would be complete without animals mindlessly going back and forth patrolling their platform?

After reading this post at Stoked Software, I like the idea of using Lua to script NPC behaviour, even though I’m sure the simple behaviours above could be much more easily done without it.  I downloaded Lua following the instructions in the second part of that post, and added it to my project by renaming the src folder lua, and dragging it into my XCode project. (Unlike the post, I downloaded Lua 5.2.1, not 5.1.)

Note this is the bare bones DIY approach, not using any existing wrappers like Wax or Corona, which are nicely described at Lua Nova. I’ll try it this way until I get frustrated.

Some initial set up decisions:

  • I decided to include the header files directly into the GameModel, since we only want one Lua state.  I was originally thinking of having one per animal.
  • Whereas the post adds an instance variable lua_State *l, I am adding a @property (readwrite) lua_State *luaState.
  • Also, to keep the NPC control separate from the rest of my code (in case I decide one day to use something other than Lua), I have added an NPC category to GameElement.

Some hurdles:

  • I get lots of “Undefined symbols for architecture i386:” errors, starting with one for luaL_error, the first Lua call I was trying to make. I posted a question on stack overflow which quickly sorted this out: it is because Box2D is in C++, but Lua is in C.  So you need to wrap the Lua includes with an extern "C" command. (See stack overflow for the precise solution.) In fact you’ll also need to wrap all the upcoming C code in this.
  • I get the error “Use of undeclared identifier ‘self‘” when the static C function which communicates with Lua tries to access the GameElement self. This is OK, the C code doesn’t have any concept of self; it means you have to pass the relevant object from Lua to C. Fortunately, the Stoked Software blog explains how to do this in Part 3, for enemy ships.
  • ARC – I have used __bridge everywhere so there is no transfer of ownership, would appreciate any thoughts on whether that’s correct.
  • Lua 5.2 does not use luaL_register, but instead luaL_setfuncs.  There’s a good discussion on the Lua users wiki, but in the end I could not make this work, so had to go back to Lua 5.1. Any advice on how to apply this to Lua 5.2 would be gratefully received.
  • The Stoked Software blog gives great examples of how to send position data from Objective C to the Lua script, and how to get the Lua script to trigger methods in Objective C so long as the only parameter is a single object.  However, I want to set a target point using Lua.  I have stumbled upon one way to do this, hinted at by the Lua users wiki, using luaL_checknumber, e.g.
    static int setTarget(lua_State *luaState) {
        //
        // Parameters: The GameElement whose target you want to set
        //             The x-coordinate of the target
        //             The y-coordinate of the target
        //
        // Returns: nothing
        //
        GameElement *element = (__bridge GameElement *)lua_touserdata(luaState, 1);
        float x = luaL_checknumber(luaState, 2);
        float y = luaL_checknumber(luaState, 3);
        NSLog(@"target (%6.2f, %6.2f) %@ %p",x,y, element.appearName, element);
        [element touchLocation:Point3DMake(x,y,0.)];
        return 0;
    }

    and then from the Lua script, it’s just:

    function process(gameElement)
      game.setTarget(gameElement, 1.1, 0.5)
    end

    I have no idea if this is a good way to do it, but it works.

     

With those hurdles surmounted, I can now use Lua to script my NPC behaviour.  I just need to think what that should be…

One last note – I see that the license for Lua requests that users give Lua credit (see the download page).

  

Controlling character movement with Box2D

So – the next challenge is a nice way to move the player’s character with Box2D. I would like the user to touch a spot in the game, and have the character try to move to that location (noting that the place on the screen may be different if the world scrolls).  The character should move at constant speed while on the ground, and follow normal physics while falling. I also like the little bounce the character gets with Box2D on landing on the ground after a fall.

For now, I am just representing the character as a ball, but ignoring its rotation for drawing.  I realise now this representation actually matters, since a ball rolls along the ground differently to how a box slides. If I need more realism, I expect to replace the ball representation with a small ball for the legs and a custom shape for the body, with a (wheel?) joint holding the two together, like a car.

Here are some alternatives:

  1. Apply an impulse to the character every update (1/60th of a second)
    I was using code like this in GameElement:

    -(void) applyImpulseToVelocity:(Point3D)targetVelocity {
        b2Vec2 vel = self.body->GetLinearVelocity();
        float dvx = targetVelocity.x - vel.x;
        float dvy = targetVelocity.y - vel.y;
        float mass = self.body->GetMass();
        self.body->ApplyLinearImpulse( b2Vec2(mass * dvx,mass * dvy),
                   self.body->GetWorldCenter() );
    }

    The target velocity is basically proportional to the distance between the character’s current location and the touch location (modified so that the character can’t fly).
    The problem is, this doesn’t look realistic at all: when the character is falling, gravity has very little effect on it – it just drifts down slowly.

  2. Use a mouse joint
    This is inspired by Ray Wenderlich’s breakout tutorial. According to the Box2D manual, a mouse joint attempts to drive a point on the body to the target. It required a bit of a rewrite of GameElement and GameModel, because the mouse joint needs to know the world and the ground body, which I was hoping to isolate from the GameElement.
    Quite an interesting effect: the character speeds to the touched location and then spins around it at a small radius as if on a rope.  Gravity does act on it at this point, if not while it speeds along. Not the game play I was after.
    Also, the manual advises “Many users have tried to adapt the mouse joint for game play. Users often want to achieve precise positioning and instantaneous response. The mouse joint doesn’t work very well in that context. You may wish to consider using kinematic bodies instead.” Hmmm… kinematic bodies, the manual says, do not respond to forces, and do not collide with static or other kinematic bodies. You normally move them by setting their velocity.  I don’t think I’ll pursue this option: I definitely want forces to apply in some situations, and want them to collide with static bodies.

  3. Apply an impulse to the character every update (1/60th of a second), but only when in contact with the ground or a platform
    This approach most closely matches the description of what I want. It felt like a hack but in fact, humans can only propel themselves when on the ground, so it is (somewhat) realistic. With the contact listener already working, this was not too hard to implement. The main work was changing the ground to a GameElement (it was previously just a part of the GameModel), and setting user data for the ground, platforms and ladders as well as animals. The contact listener needs the user data to send a begin/end contact message to the object.
    This is virtually the same as option 1 above, but is only applied if a flag is set on the character’s GameElement saying that it is in contact with the ground, a platform or a ladder.  In fact all these are subclasses of the Platform class, so the test is very simple.
    Great news – this works pretty well!

Nonetheless I expect this is not the end of the story…

  

Saving objects as property lists

Previously I have saved objects in Cocoa using NSEncoding.  However, for my game I want to save the GameModel’s layout as an intelligible text file.

One way to do this is to save the object as a property list (or plist), which is simply an NSDictionary.  The result is an XML file.

My game has a GameModel class and this has an NSArray of GameElements. So to pull this off, I added two methods to each of these classes.  The first returns a dictionary which describes the layout of the object.  For GameElement, it looks like this:

-(NSDictionary*) layoutDictionary {
  //
  // Returns keys and values of all the parts of the object 
  //    that need to be saved to define this game layout.
  // This only needs to be overridden by subclasses 
  //    if special structs need to be added (e.g. Point3D).
  // Apart from this, instead, subclasses should add to layoutKeys.
  //
  NSDictionary* dict1 = [self dictionaryWithValuesForKeys:self.layoutKeys];
  NSMutableDictionary* dict = [NSMutableDictionary dictionaryWithDictionary:dict1];
  dict[@"where"] = NSStringFromPoint3D(self.where);
  dict[@"class"] = NSStringFromClass([self class]);
  return dict;
}

It uses the key-value coding (KVC) method dictionaryWithValuesForKeys, which returns the values of the object’s own methods or variables.  I declare which properties are important to save in the layoutKeys variable, which I set up in the GameElement initialisation.

However… you can see there is some special handling of the where struct above.  The reason is that while an NSDictionary can happily have any object in it, if you want to write it out as a plist file, you must only have “plist objects” – NSData, NSString, NSNumber, NSDate, NSArray, NSDictionary and the like. If “where” was in the layoutKeys, the first line above would happily add an NSValue object containing the where struct’s data.  But you could not write this out.

It would be great if you could add custom plist objects.  Please let me know if you know how to do this.  In the absence of this, I have written functions to convert the struct into a plist object (an NSString), and back again, instead. For CGPoint these functions already exist.

The second function then creates an object from a dictionary, for example:

+(GameElement*) elementWithLayoutDictionary:(NSDictionary*)dict inModel:(GameModel*)model {
    //
    // Creates an element, given the relevant piece of the game layout dictionary, and the model.
    // This needs to be overridden by subclasses
    //
    Point3D where = Point3DFromNSString([dict valueForKey:@"where"]);
    GameElement* element = [[self class] elementNamed:dict[@"name"] at:where inModel:model];
    element.scale = [[dict valueForKey:@"scale"] floatValue];
    return element;
}

The GameModel also has layoutDictionary and modelWithLayoutDictionary: functions, which loop through the array of GameElements, e.g.

+(GameModel*) modelWithLayoutDictionary:(NSDictionary*)dict {
    int elementNumber = 1;
    while (NSDictionary* elementDict = [dict valueForKey:
                [NSString stringWithFormat:@"Element-%d", elementNumber]]) {
        [self addElement:[NSClassFromString(elementDict[@"class"]) 
                elementWithLayoutDictionary:elementDict inModel:self]];
        elementNumber++;
    }
}

(You can see that this implementation allows for subclassing of the GameElements.)

That’s basically it.  I then write this out to a plist file using
[layoutDict writeToFile:path atomically:YES];
and read it in using
[NSDictionary dictionaryWithContentsOfFile:path].

I suspect this could be improved:

  • KVC allows for a one-to-many relationship.  Can I use this to remove my custom handling of the array?
  • Is there a way to remove the custom handling of non-plist-objects like the Point3D struct?
  • Perhaps this all comes for free if I used CoreData… but do you then lose control of the data format?

Any thoughts?

  

Cocos2D, Box2D and MVC

As I indicated in my last post, I am going to write my kids’ game in Cocos2D. After implementing the simple bear animation tutorial from Ray Wenderlich, I had a few questions:

  1. How do you use automatic reference counting (ARC) with Cocos2D? The approach I’ve taken is to follow this blog by Steffen Itterheim.
  2. How do you write code with Cocos2D (and maybe with Box2D) in an MVC (model-view-controller)-compliant way?  The approach I’ve taken is to follow this blog and its sequel by Bartek Wilczyński.
  3. How do you add Box2D to the project once you’ve already started it (and continue to use ARC)?  The approach I’ve taken is to make it a static library by following this blog by Red Glasses. This is a pretty involved procedure but does the job. I imagine you can do point (1) above using this approach too – it would be nice to be consistent – but I have not tried this.

I’ve done all this and got it all running, and I thought others may find it helpful, so I have put the result on github at https://github.com/RacingTadpole/Cocos2D-Box2D-MVC-example. You are welcome to download it as a starting point for your own projects, or just to poke holes in it (but please tell me what they are!). The frame rate seems to be only 20-30 frames per second on the simulator, but on a real device it is close to 60, and I have read elsewhere that this stat should be ignored on the simulator anyway. Edit – the project on GitHub was compiled using a slightly older version of XCode, and the latest XCode complains when I try to run it on a device (ld: file is universal (2 slices) but does not contain a(n) armv7s slice: …./libbox2d-lib.a for architecture armv7s ). For now I am solving this by setting “Build Active Architecture Only” to “Yes” in the target’s build settings (see this stackoverflow question), but in the long run it looks like I’ll need to redo step 3 above using the latest XCode.

The game as it stands (called Zambazi) simply involves a host of monkeys and bears falling from the sky onto a grassy foreground, and bouncing like rubber balls. When you touch anywhere, all the animals are hit with random forces.  It’s not much, but my kids find it surprisingly engrossing! They decided you win if you can make all the bears run off the screen before the monkeys… Maybe I’m not so far from the App Store after all? :-)

All images are from Ray & Vicki Wenderlich’s sites – thank you both for making these freely available!

Here’s the basic structure/flow:

GameController

When PLAY is pressed, a GameController is initialized.  The gameController has a GameView and a GameModel, each of which is initialized. The gameController also schedules the updates.

GameView

The gameView has a delegate (actually the gameController) which currently does nothing, because it doesn’t need to know which sprite you’ve touched. This delegate may be useful if you do need to know – see Bartok’s blog for his vision here. In fact I’m planning instead to remove this delegate and simply register the controller with Cocos2D as a touch delegate, as described here.

The view creates the necessary layers:

GameLayer

The gameLayer is in charge of all the sprites. It knows nothing about the view or the controller, but does have a reference to the model. It also registers itself as an observer of notifications from the model. The notifications are:

  • Model initialization – this is so that the gameLayer can get a reference to the model in the first place.
  • Revise game elements – this is so that it can set up the sprites that correspond to the model.

The gameLayer starts by loading in all the sprites for the game, and setting up their actions (CCAction).  I am sure there is a much better way to do this – but this does the job for now.

It also keeps track of which actions are running. This seems an unfortunate complication, but as far as I know, you need to do this so that you can stop the action later. If you can get away with stopping all actions on a sprite – and maybe you can – then you could remove this stuff and use stopAllActions instead.

The gameLayer does not know about Box2D – I see that as a model-level thing.  The gameElements provide their own velocity, where, rotation etc methods.  I have defined a Point3D structure to pass around points – this was when I was thinking the model may have a 3D world even if the view is only 2D.  However, with Box2D, the third dimension is irrelevant – so it would be simpler to just use CGPoint for example.  I am leaving Point3D nonetheless so that if you want to use this code with a 3D model (without Box2D), it shouldn’t be too hard to adapt it.

The other trick is that gameLayer has an NSDictionary (called sprites), with the gameElements as keys and the sprites as the objects. The complication is that NSDictionary copies its keys before it uses them, so that the key winds up being a different object to what you requested.  The solution (implemented in my code) is to override copyWithZone: to return self, without copying, as described in this stack overflow post.  I am assured this is good practice if the object is immutable. All this may be too tricky by half, but it seemed sensible at the time, and works fine.

GameModel

The game model is the Box2D world, and an array of the game elements. When createGameObjects is called (by the Controller – this could equally well be part of the initialization), it sets up some default game elements. It has an update: method which uses Box2D to update the physics; optionally each element may have its own additional update behaviour (I have adopted Bartok’s “Updatable” protocol).

GameElement

The gameElements are the models of the platforms, the enemies, etc.  They basically have a Box2D body and a name.  The name is used by the gameLayer to work out what sprite to show. The gameElement uses Box2D to provide where, velocity and rotation methods, so the body variable itself is kept private (i.e. in a class extension).

I am subclassing GameElement (e.g. Animal) to provide different behaviour for different models.

Technical note – so that subclasses can still access the body variable, I have declared body in a class extension header file called GameElements_Private.h.  Then GameElements.m and Animal.m both import GameElements_Private.h instead of GameElements.h, so that they can refer to body.

Further ideas

As I start to turn this into a functioning game, I have found two further issues, one conceptual, one practical:

  • It’s nice to cleanly separate the model from the view – but the image you are using is a particular size, and I’m finding the model sometimes needs to know this size (so you don’t have to scale the image). I’m solving this with another delegate pattern.  I’ve introduced a NaturalSizeProtocol, which the GameLayer and the GameView follow.  The GameModel then has a naturalSizeDelegate.  When a gameElement needs to know its natural size (i.e. the size the view wants to make it), it asks its naturalSizeDelegate.  This returns the size in model co-ordinates (as a Point3D).  This feels like a contortion of MVC, so I’d love to hear if anyone has a better solution to this.  It has left me wondering if MVC is more trouble than it’s worth after all for image-intensive games.
  • Getting a background image to repeat in Cocos2D is hard.  I have only managed it so far by loading the background image multiple times, which doesn’t seem right.

That’s it for now.  Please let me know if you find this useful, or have any suggestions.

Edit – I have just come across Steffen Itterheim’s excellent post on exactly this subject, which inspired him to write KoboldTouch. Together with problems I am having getting Lua to compile in my Cocos2D/Box2D project, I am starting to wish I had used Kobold2D…

  

A kids’ game

I have an idea for a kids’ game (though there’s quite a bit of competition out there on the App Store!).  The question I’m debating is whether I should implement it as a 2D game or a 3D game. It would only have a fixed camera position, but I’d like some depth to the world, so that characters going up the screen go into the distance, and can then go behind other characters or objects. From Wikipedia’s 2.5D entry, this could be considered “scaling along the z-axis”. I actually haven’t found any games with that sort of view, perhaps because a 2D model looks dodgy (see below), and by the time you have a 3D model, there are so many cooler things you can do with one. Here are the options I’m considering:

2-dimensional graphics

This basically comes down to Cocos2D. I put some code together using Cocos2D for iOS, based on Ray Wenderlich’s tutorial for an animated bear, and made the bear smaller if it goes up the screen and bigger if it comes to the front.  This looks pretty good and is quite simple to do, as long as the bear is not going mostly up or down the screen.  In that case you’d need more animation sequences so the bear faces towards or away from you.  But what if you go on a diagonal away from you or towards you?  I think you’d need those sequences too.  That’s quite a few sequences, and of course whereas any walking angle would be allowed, graphically it would have to snap to one of the 8 available animations. So the problems with this approach are:

  1. I couldn’t use freely available art (like Vicki Wenderlich has available)
  2. It won’t look awesome – and I’m not sure it will even look good enough.

Also, no 2D physics engines (like those that come with Cocos2D) would work, since the gameplay is not actually 2D. In my case I don’t plan to have any complex physics, so I can make my own.  Potentially though I could use a 3D physics library like Bullet.

3-dimensional graphics

I think this basically comes down to Unity3D (and perhaps Unreal, but I can’t use this on a Mac so it is less appealing), although there are a few other related options out there too. Unity3D is amazing, there are lots of 3D graphical assets available on their asset store, and compiles cross-platform to boot. I have succeeded in building my own terrain in its editor, putting in an eerie sky and swaying grass, and can compile a game to my iPad which lets me look around it (in a first person view).  I followed a youTube demo for this from TechZone. I have also read about ways to layer native iOS elements on top of this – Millipede gives a good explanation of this here, and their Navy Sink’Em game is the best example I’ve found of the camera angle I want, though I didn’t intend that level of realism. Adding a iOS front-end to a Unity project is also covered by this Blurst post. The problems with a Unity3D approach are:

  1. Troubleshooting how to do things in Unity is probably harder than I’m used to with Cocoa.
  2. Will take some learning how to communicate between Unity and iOS.
  3. I can imagine the project flow getting quite complex as I want the player to be able to add assets to the scene, and associate scripts with them. I probably want an iOS interface to do this, which would need to display Unity assets.
  4. Costs a bit.
  5. I don’t like being forced to use a new development environment, and being locked into Unity’s world.

The other 3D options include:

Cocos3D. There is very little online about this – no tutorials. It also comes with the caveat that it “is not yet compatible with cocos2d 2.x. Please use cocos2d 1.x.”

SIO2. I have just come across this here. It looks promising, as it does not force you into a particular SDK.  I have not investigated this in detail.

Conclusion

I think I’m biting off more than I can chew right now if I try to make a grand 3D game. So I will adopt the 2D framework for now and try to get a good game logic happening.  Perhaps even give up and modify the game flow so that it is a true 2D game (with parallax of course… I always loved Moon Patrol).  Then, if that doesn’t look good enough to release, come back to Unity.

Any advice?

  

Turning on the Google App Engine

When it comes to deploying a python script on the web, Google App Engine is one approach.  It took me a while to work out how to get started on this, but here is the way forward if you already have a domain site registered with Google Apps. Until I did this, I repeatedly got an “Unauthorized” message when I went to the AppEngine page.

  • log in as the admin for your site, e.g. at mail.yourdomainname.com
  • go to the control panel, www.google.com/a/yourdomainname.com
  • choose Organizations and Users, and the Services tab within that
  • turn Google App Engine on, and save changes
  • go to www.appengine.com/a/yourdomainname.com
  

Access MAMP’s MySQL database from Python

I posted a while ago about running a locally hosted server using MAMP, and I have been successfully using a PHP script on this server to read and write to a database.

Then a friend suggested I could use Python instead of PHP, e.g. using “CherryPy” as a light-weight server.  I am a big fan of python, so I want to try this.

The basic overview is:

  • Use the CherryPy python library to run the server (though more recently I have switched to Django)
  • Use the MySQLdb python library to interface with MySQL
  • Use the MAMP-installed instance of MySQL

Here’s how I made it work:

    • Download CherryPy (or Django or any other Python-based framework) from the link above.  I had to use sudo in front of the python setup.py build command.
    • Download MySQLdb from the link above.
    • Check it works by going to the terminal, typing python and then import MySQLdb
    • It doesn’t work, for several reasons:
      • It can’t find the “mysql_config” file associated with MAMP.  To fix this, open site.cfg (in the directory you installed MySQLdb) and add the line below. Note this points to the MAMP installation of my_sql, not the default of /usr/local/bin:
          mysql_config = /Applications/MAMP/Library/bin/mysql_config
      • It couldn’t find llvm. To solve this, open XCode, go to Preferences, choose the Components tab, and install the command line tools.
      • MAMP does not include the required .h files.  This post describes how to get around this: I downloaded the 64bit MySQL from here, and copied files from its include directory into the new directory MAMP/Library/include, and from its lib directory into MAMP/Library/lib. (Don’t use the 32bit files.)
      • The final magic touch is you need to type
          export DYLD_LIBRARY_PATH=/Applications/MAMP/Library/lib
        into the terminal before you go into python. (But note – this will mess up git.  So when you need git, type export DYLD_LIBRARY_PATH=''. Urrggh!)
    • Now both import MySQLdb and import _mysql work in python.
    • To access MySQL from the terminal, open MAMP and start the servers.  Then type
        /Applications/MAMP/Library/bin/mysql --host localhost -uroot -proot
      You can now do things like show databases; to look at the databases you have available.
    • So far so good, but when I try to connect to a database, I got the error “Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)” (error number 2002).  The solution is nicely explained in this post, and is:
        sudo ln -s /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock
    • Strangely, when I came back to this a week later, I got an error: “Library not loaded: libmysqlclient.18.dylib”.  I found this works:
        sudo ln -s /Applications/MAMP/Library/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib
      I also found I had to repeat the previous point to recreate the /tmp/mysql.sock file.

To summarise the on-going usage:

  • Before using python, you need to type export DYLD_LIBRARY_PATH=/Applications/MAMP/Library/lib
  • In order to use MySQL in the terminal, you need to type /Applications/MAMP/Library/bin/mysql --host -localhost -uroot -proot

In Django, you will need to use the following DATABASE values in settings.py:

    'HOST': '/Applications/MAMP/tmp/mysql/mysql.sock',
    'PORT': '8888',

One more thing. When I write apps that interface with the MAMP localhost, I need to replace localhost with the local IP address (from the Mac Network Utility program), and append port :8888.  However the CherryPy localhost has the local IP address 127.0.0.1 (and  :8080). Why are the two local IP addresses different?

From here:

Any thoughts?