Category Archives: Websites

Experience with Meteor

Meteor Day is upon us!

Here are the slides I am going to present on my experience developing with Meteor. In a nutshell -

  • Love the principles – they made it easy to develop a great webapp quickly
  • I’ve done a few cool things, I think! (like encryption, undo, admin panel with datatables, custom art:accounts-ui)
  • But… SEO is killing me (only two keywords showing!?)
  • The Paypal IPN has been a pain – still unresolved
  • Can’t escape the usual browser issues – and Iron Router adds a few for IE9
  • Speed has been an intermittent problem
  • Still feeling my way towards a good programming style
  • Please check out the app!
(Just click on the slide above and then you can advance through the deck using the arrow keys. Alternatively, click here to see them in a new tab.)


You can see the slides from the talk online here.

  

Harness your Python style to write good Javascript

Python encourages you to write good code. Javascript does not. How can you harness your Python style to write good Javascript?

Classes

Let’s try to write this Python code in Javascript.

class Vehicle(object):
    def __init__(self, size):
        self.size = size
    def __str__(self):
        return "Vehicle of size {0}".format(self.size)

class Hovercraft(Vehicle):
    def __init__(self, *args, **kwargs):
        # initialize just like a vehicle
        super(Hovercraft, self).__init__(*args, **kwargs)
        # add extra custom init if needed
    def hover(self, height):
        print("Hovering at height {0}".format(height))

# eg. make a new size 8 hovercraft
h = Hovercraft(8)
# eg. hover at height 12
h.hover(12)
# eg. change size and print
h.size = 9
print(h)

Not obvious? The problem is that Javascript doesn’t have an out-of-the-box analogue to classes. Somehow we need to adapt objects and functions to the task.

Use prototypes

Your first thought might be to define new object types by defining object constructors, which are called with the new keyword, and adding methods to the object’s prototype chain. These objects look a lot like classes, don’t they? So you might write something like this:

// Not the best approach - see below
function Vehicle(size) {
    this.size = size;
    // or use this and arguments pseudo-arguments
}
Vehicle.prototype.toString = function () {
    return "Vehicle of size " + this.size;
};
function Hovercraft(size) {
    this.size = size;
}
Hovercraft.prototype = new Vehicle();
Hovercraft.prototype.hover = function (height) {
    alert("hovering at height "+height);
}

// eg. make a new size 8 hovercraft
h = new Hovercraft(8);
// eg. hover at height 12
h.hover(12);
// eg. change size and print
h.size = 9
h.toString()

This has a number of problems:

  • Vehicle and Hovercraft have the same constructor, but you can’t reuse it.
  • It feels strange to define the class’s methods outside the constructor, with the prototype lines.
  • There’s no room for private variables or methods.

Use closures

For a better solution, we need to think laterally – and use Javascript’s closures. I was hesitant to make use of these at first, thinking it would only lead to perverse and impenetrable code. But in fact they are a force for good, as we’ll see.

Put simply, a closure is a function which has variables bound to it. You write the closure as a function inside another function. The trick is that the inner function can refer to any of the outer function’s local variables. (Thanks for this succinct summary StackExchange!)

Here’s a nice example of a closure from a talk on functional programming given by Douglas Crockford, JavaScript architect at PayPal and formerly Yahoo (at 57mins). (In fact the previous example is from this talk too.) In this example, a closure is used to produce an object called singleton, which has a private variable, a private function, and two methods. You call the two methods as singleton.firstMethod(a,b) and singleton.secondMethod(c):

var singleton = (function () {
    var privateVariable;
    function privateFunction(x) {
        ...privateVariable...
    }

    return {
        firstMethod: function (a, b) {
            ...privateVariable...
        },
        secondMethod: function (c) {
            ...privateFunction()...
        }
    };
}() );
// note the function is called immediately,
// so the var singleton is its returned value
// the surrounding brackets are just to help the reader

So let’s adapt that to our problem:

function vehicle(size) {
    var that = {
        size: size,
        toString: function () { 
            return "Vehicle of size " + this.size; 
        },
    };
    return that;
}
function hovercraft(size) {
    var that = vehicle(size);  // inherit from vehicle
    that.hover = function(height) { 
        alert("hovering at height "+height);
        return that; // optional - allows chaining
    };
    return that;
}

// eg. make a new size 8 hovercraft
h = hovercraft(8);
// eg. hover at height 12
h.hover(12);
// eg. both at once using chaining
g = hovercraft(8).hover(12);
// eg. change size and print
h.size = 9
h.toString()

In the above, size is accessible to the world, just as it is in Python. But Javascript also lets us make it private:

function vehicle(size) {
    // can define private variables here
    // instead, here we use the fact that parameters are private
    return {
        toString: function () { 
            return "Vehicle of size " + size; 
        },
    };
}

In general, to make your own constructor function (eg. vehicle, hovercraft), you follow this recipe from the talk:

  1. Make an object
  2. Define (private) variables and functions
  3. Augment the object with methods (which have access to the privates above)
  4. Return the object

This pattern has a name: the module pattern. You’ll find a great writeup of it, and some ways to use it across multiple js files, in this post by Ben Cherry. He also points out the best way to handle dependencies, and to update existing variables.

Use chaining

In the example above, I have added the extra feature of “chaining”. In Python you have to put each effect on a separate line, eg.:

h.size = 9
h.hover(18)
print(h)

But in Javascript you can potentially chain it all together into one, like so:

h.size(9).hover(18).toString()

I first discovered the joy of chaining while using Mike Bostock’s super-powerful D3 library. He describes how to do it here – in fact, his description of how to write a reusable chart arrives at the same closure-based solution as we have, with the addition of getters and setters as below.

We made hover chain, but size do it doesn’t yet, as it’s a variable, not a function. Mike solves this by adding a getter/setter function for each public variable, which we could add like this:

function vehicle(startSize) {
    var size = startSize;
    var that = {
        toString: function () { 
            return "Vehicle of size " + size;
        },
    };
    // getter/setter functions
    that.size = function(_) {
        if (!arguments.length) return size;
        size = _;
        return that; // Q: would 'return this' work?
    };
    return that;
}
// eg. this works now
h.size(9).hover(18).toString()

Then h.size() returns (gets) the hovercraft h‘s size, and h.size(9) sets the size to 9.

Another benefit: the code never refers to this. That’s handy, because I find whenever I refactor code into smaller functions I get tripped up by the meaning of this changing.

You may also recognize such getters and setters from jQuery, where eg. $("body").text() returns the page body’s text, and $("body").text("eels") sets the body’s text to “eels”.

Still, as nice as chaining is, needing to add 5 lines of boilerplate code for every variable, with 3 references to the variable name that must be changed each time, is the sort of thing we became programmers to avoid.

To solve this, I am starting to put all the variables with getters and setters into a single object, eg. xt:

function vehicle(startSize) {
    var ext = {size: startSize};
    if (Object.keys) {
        var extKeys = Object.keys(ext);
    }
    function toString() {
        return "Vehicle of size " + ext.size;
    }
    return {
        toString: toString,
        get: function(name) {
            if (!arguments.length) return ext;
            return ext[name];
        },
        set: function(name, val) {
            if (typeof extKeys!=="undefined" && extKeys.indexOf) {
                if (extKeys.indexOf(name)>=0) {
                    ext[name] = val;
                } else {
                    throw Error("Variable "+name+" not found");
                }
            } else {
                // on browsers without Object.keys or indexOf,
                // don't check the name is valid
                ext[name] = val;
            }
            return this;
        }   
    }
}
// eg. these work
h.get("size");
h.set("size",9).hover(18).toString();

Don’t make it a global

Finally, you probably don’t want to have a new global called vehicle. It’s better to add it to another module, eg. RT (which may or may not already exist), as RT.vehicle. It might also depend on other modules, eg. the underscore (_) library. To do this, wrap the whole function in another closure!

if (typeof _ === 'undefined') { 
    throw new Error('Vehicle requires underscore') 
}
(function(RT, _) {
    function vehicle(startSize) { 
        ... // copy from above
    }

    // attach vehicle to RT
    if (typeof RT==="undefined") {
        RT = {};
    }
    RT.vehicle = vehicle;
    return RT;
}(typeof RT === "undefined" ? {} : RT, _));
// eg. make a new size 9 vehicle
v = RT.vehicle(9);

Too crazy?

In conclusion

Javascript’s closures give you access to some interesting programming patterns. Foremost among them, it lets you implement Python-style classes, with the added bonus of private variables and functions. And this is not just an academic gimmick that risks complicating your code in the real world: it is championed by the people who develop javascript, and it is used by jQuery and D3 among others. It helps you to write good, reusable code.

So – please let me know if you’ve used this pattern before, and whether my comparison to Python’s classes stacks up.

A final thought – perhaps it is wrong to compare Javascript to Python after all. Perhaps it is better compared to LISP!

  

Nicely format tables without using table-layout:fixed

I have recently helped to develop Signup Zone, a very handy app where people can easily create their own signup sheets and rosters. People have started using it for interesting purposes, including registering to help injured wildlife! And this has led to people entering in unexpected data, such as very long email addresses; and creating sheets with a lot of columns.

At its heart, the signup sheet is simply an HTML table such as this:

Date Name Contact email address
10/11/2014 Racing Tadpole racingtadpole@example.com

This table has the code:

<table>
    <thead>
        <tr>
            <th scope="col">Date</th>
            <th scope="col">Name</th>
            <th scope="col">Contact email address</th>
        </tr>
    <tbody>
        <tr>
            <td>10/11/2014</th>
            <td>Racing Tadpole</th>
            <td>racingtadpole@example.com</th>
        </tr>
    </tbody>
</table>

Since I don’t know in advance how to size the different columns, I actually really like the browser’s built-in ability to choose its own column widths.

However, the browser will not let any content spill outside a cell, and it only breaks words at white space. So if one user enters a long email address (or URL) into a cell, that column becomes very wide and the table either spills outside the page margins or looks unsightly – or both, like this:

Date Name Email
10/11/2014 Racing Tadpole racingtadpole@example.com
11/12/2014 Another Tadpole anotherracingtadpole@quitelong.verylong.superlong.megalong.example.com

Here are two solutions:

  • Use the new css command hyphens: auto. It won’t work in Opera, Chrome (!) or older browsers (eg. IE 9), and it can hyphenate too many words, but it looks ok, eg:
    Date Name Email
    10/11/2014 Racing Tadpole racingtadpole@example.com
    11/12/2014 Another Tadpole anotherracingtadpole@quitelong.verylong.superlong.megalong.example.com
  • Run some javascript to insert a “zero-width space” or an invisible soft hyphen in places where the offending text can break, eg. at the . or @ symbols. This gives you control over where the words break, eg:
    Date Name Email
    10/11/2014 Racing Tadpole racingtadpole@​example​.com
    11/12/2014 Another Tadpole anotherracingtadpole@​quitelong​.verylong​.superlong​.megalong​.example​.com

    However, if someone copies the email address it will also copy the invisible characters, which could cause trouble.

Here’s some javascript code to implement the second solution in Meteor:

<template name="example">
    ...
    <td>{{{email}}}</td>
    ...
</template>
function htmlEscape(str) {
    return String(str).replace(/&/g, '&amp;')
            .replace(/</g, '&lt;')
            .replace(/>/g, '&gt;')
            .replace(/"/g, '&quot;')
            .replace(/'/g, '&#39;');
}

function myMarkup(str) {
    return String(str).replace(/@/g, '&#8203;@')
            .replace(/\./g, '&#8203;.')
            .replace(/\//g, '&#8203;/');
}

Template.example.email = function(){
    // suppose s is the string we want to show
    return myMarkup(htmlEscape(s));
}

The htmlEscape function is inspired by this Stack Overflow post. An alternative is to use Spacebars.SafeString.

Hope that’s helpful!

  

9 Lessons from PyConAU 2014

A summary of what I learned at PyCon AU in Brisbane this year. (Videos of the talks are here.)

1. PyCon’s code of conduct

Basically, “Be nice to people. Please.”

I once had a boss who told me he saw his role as maintaining the culture of the group.  At first I thought that seemed a strange goal for someone so senior in the company, but I eventually decided it was enlightened: a place’s culture is key to making it desirable, and making the work sustainable. So I like that PyCon takes the trouble to try to set the tone like this, when it would be so easy for a bunch of programmers to stay focused on the technical.

2. Django was made open-source to give back to the community

Ever wondered why a company like Lawrence Journal-World would want to give away its valuable IP as open source? In a “fireside chat” between Simon Willison (Django co-creator) and Andrew Godwin (South author), it was revealed that the owners knew that much of their CMS framework had been built on open source software, and they wanted to give back to the community. It just goes to show, no matter how conservative the organisation you work for, if you believe some of your work should be made open source, make the case for it.

3. There are still lots more packages and tools to try out

That lesson’s copied from my post last year on PyCon AU. Strangely this list doesn’t seem to be any shorter than last year – but it is at least a different list.

Things to add to your web stack -

  • Varnish – “if your server’s not fast enough, just add another”.  Apparently a scary scripting language is involved, but it can take your server from handling 50 users to 50,000. Fastly is a commercial service that can set this up for you.
  • Solr and elasticsearch are ways to make searches faster; use them with django-haystack.
  • Statsd & graphite for performance monitoring.
  • Docker.io

Some other stuff -

  • mpld3 – convert matplotlib to d3. Wow! I even saw this in action in an ipython notebook.
  • you can use a directed graph (eg using networkx) to determine the order of processes in your code

Here are some wider tools for bioinformaticians (if that’s a word), largely from Clare Sloggett’s talk -

  • rosalind.info – an educational tool for teaching bioinformatics algorithms in python.
  • nectar research cloud – a national cloud for Australian researchers
  • biodalliance – a fast, interactive, genome visualization tool that’s easy to embed in web pages and applications (and ipython notebooks!)
  • ensembl API – an API for genomics – cool!

And some other sciency packages -

  • Natural Language Toolkit NLTK
  • Scikit Learn can count words in docs, and separate data into training and testing sets
  • febrl – to connect user records together when their data may be incorrectly entered

One standout talk for me was Ryan Kelly’s pypy.js, implementing a compliant and fast python in the browser entirely in javascript. The only downside is it’s 15 Mb to download, but he’s working on it!

And finally, check out this alternative to python: Julia, “a high-level, high-performance dynamic programming language for technical computing”, and Scirra’s Construct 2, a game-making program for kids (Windows only).

4. Everyone loves IPython Notebook

I hadn’t thought to embed javascript in notebooks, but you can. You can even use them collaboratively through Google docs using Jupyter‘s colaboratory. You can get a table-of-contents extension too.

5. Browser caching doesn’t have to be hard

Remember, your server is not just generating html – it is generating an http response, and that includes some headers like “last modified”, “etag”, and “cache control”. Use them. Django has decorators to make it easy. See Mark Nottingham’s tutorial. (This from a talk by Tom Eastman.)

6. Making your own packages is a bit hard

I had not heard of wheels before, but they replace eggs as a “distributable unit of python code” – really just a zip file with some meta-data, possibly including operating-system-dependent binaries. Tools that you’ll want to use include tox (to run tests in lots of different environments); sphinx (to auto-generate your documentation) and then ReadTheDocs to host your docs; check-manifest to make sure your manifest.in file has everything it needs; and bumpversion so you don’t have to change your version number in five different places every time you update the code.

If you want users to install your package with “pip install python-fire“, and then import it in Python with “import fire“, then you should name your enclosing folder python_fire, and inside that you should have another folder named fire. Also, you can install this package while you are testing it by cding to the python-fire directory and typing pip install -e . (note the final full-stop; the -e flag makes it editable).

Once you have added a LICENSE, README, docs, tests, MANIFEST.insetup.py and optionally a setup.cfg (to the python-fire directory in the above example) and you have pip installed setuptoolswheel and twine, you run both

python setup.py bdist_wheel [--universal]
python setup.py sdist

The bdist version produces a binary distribution that is operating-system-specific, if required the universal flag says it will run on all operating systems in both Python 2 and Python 3). The sdist version is a source distribution.

To upload the result to pypi, run

twine upload dist/*

(This from a talk by Russell Keith-Magee.)  Incidentally, piprot is a handy tool to check how out-of-date your packages are. Also see the Hitchhiker’s Guide to Packaging.

7. Security is never far from our thoughts

This lesson is also copied from last year’s post. If you offer a free service (like Heroku), some people will try to abuse it. Heroku has ways of detecting potentially fraudulent users very quickly, and hopes to open source them soon. And be careful of your APIs which accept data – XML and YAML in particular have scary features which can let people run bad things on your server.

8. Database considerations

Some tidbits from Andrew Godwin’s talk (of South fame)…

  • Virtual machines are slow at I/O, so don’t put your database on one – put your databases on SSDs. And try not to run other things next to the database.
  • Setting default values on a new column takes a long time on a big database. (Postgres can add a NULL field for free, but not MySQL.)
  • Schema-less (aka NoSQL) databases make a lot of sense for CMSes.
  • If only one field in a table is frequently updated, separate it out into its own table.
  • Try to separate read-heavy tables (and databases) from write-heavy ones.
  • The more separate you can keep your tables from the start, the easier it will be to refactor (eg. shard) later to improve your database speed.

9. Go to the lightning talks

I am constantly amazed at the quality of the 5-minute (strictly enforced) lightning talks. Russell Keith-Magee’s toga provides a way to program native iOS, Mac OS, Windows and linux apps in python (with Android coming). (Keith has also implemented the constraint-based layout engine Cassowary in python, with tests, along the way.) Produce displays of lightning on your screen using the von mises distribution and amazingly quick typing. Run python2 inside python3 with sux (a play on six).  And much much more…

Finally, the two keynotes were very interesting too. One was by Katie Cunningham on making your websites accessible to all, including people with sight or hearing problems, or dyslexia, or colour-blindness, or who have trouble with using the keyboard or the mouse, or may just need more time to make sense of your site. Oddly enough, doing so tends to improve your site for everyone anyway (as Katie said, has anyone ever asked for more flashing effects on the margins of your page?). Examples include captioning videos, being careful with red and green (use vischeck), using aria, reading the standards, and, ideally, having a text-based description of any graphs on the site, like you might describe to a friend over the phone. Thinking of an automated way to do that last one sounds like an interesting challenge…

The other keynote was by James Curran from the University of Sydney on the way in which programming – or better, “computational thinking” – will be taught in schools. Perhaps massaging our egos at a programming conference, he claimed that computational thinking is “the most challenging thing that people do”, as it requires managing a high level of complexity and abstraction. Nonetheless, requiring kindergarteners to learn programming seemed a bit extreme to me – until he explained at that age kids would not be in front of a computer, but rather learning “to be exact”. For example, describing how to make a slice of buttered bread is essentially an algorithm, and it’s easy to miss all the steps required (like opening the cupboard door to get the bread). If you’re interested, some learning resources include MIT’s scratch, alice (using 3D animations), grok learning and the National Computer Science School (NCSS).

All in all, another excellent conference – congratulations to the organisers, and I look forward to next year in Brisbane again.

  

Make an animated & reusable barchart

Do you need a dynamic bar chart like this in your web page, with positive and negative values, and categories along the x-axis?

Dog breeds are on my mind at the moment as we just bought a new Sheltie puppy – this chart might show a person’s scores for each breed. Click the button above to see the next person’s (random) set of scores.

This is an example of a reusable chart built using d3. Using it is fairly simple, eg.:

<script src="http://d3js.org/d3.v3.min.js"></script>
<script src="barChart.js"></script>
<script>
    var points = [['Beagle',-10],
                  ['Terrier',30],
                  ['Sheltie',55],
                  ['Greyhound',-24]];
    var myChart = d3.elts.barChart().width(300);
    d3.select("body").datum(points).call(myChart);
</script>

Please check out the source code for barChart.js on bitbucket.

You may also find this helpful even if you don’t need a barchart, but want to understand how to build a reusable chart. I was inspired when I read Mike Bostock’s exposition, Towards reusable charts, but I found it took some work to get my head around how to do it for real – so I hope this example may help others too.

The key tricks were:

  • How to adjust Mike Bostock’s sample code to produce bars instead of a line and area
  • Where the enter(), update and exit() fit in (answer: they are internal to the reusable function)
  • How to call it, including whether to use data vs datum (answer: see code snippet above)
  • How to refer to the x and y coordinates inside the function (answer: the function maps them to d[0] and d[1] regardless of how they started out)

You can find a much fancier version of this chart, with a range slider and hover text, in this post.

Good luck!

  

Hosting Meteor with MongoDB on Webfaction

Meteor is constantly evolving, and so are the ways to host it. I couldn’t find any single source that explained how to deploy a Meteor site to Webfaction using the latest technology (ie. as of March 2014). So here is how I did it for my site, Signup Zone.

First, follow Webfaction’s instructions to install a MongoDB app and add a data directory. I have called the app mongodb1. Note down its port number, eg. 18000.

The instructions say to add a user with the ["userAdminAnyDatabase"] role. I have also added a second user with the roles ['clusterAdmin', 'userAdminAnyDatabase', 'readAnyDatabase'], based on this Stackoverflow answer.

The instructions then say to run this in an SSH window:

./mongodb-linux-x86_64-2.4.9/bin/mongod --auth --dbpath $HOME/webapps/mongodb1/data/ --port 18000

Use Webfaction’s one-click installer to install a node.js app, and note down its port number too, eg. 17000. Upgrade it to the latest stable version using the “n” package, following Webfaction’s instructions for node.

Following this post, I demeteorized my meteor app, pushed it up to a git repo on Webfaction, and pulled it down into $HOME/webapps/nodeapp/demeteoredapp/.

As mentioned in the Webfaction instructions, in the nodeapp directory, type export PATH=$PWD/bin/:$PATH. Then cd demeteoredapp and type npm install in that directory.

In another SSH shell, enter (the DB name is “admin”):

export MONGO_URL="mongodb://mongo_user:mongo_password@localhost:18000/admin?autoReconnect=true"  # 2
export MAIL_URL='smtp://user:password@smtp.webfaction.com'
export PORT="17000"
export ROOT_URL="http://example.com"

Then type:

./bin/node $HOME/webapps/nodeapp/demeteoredapp/main.js

You can adjust the default bin/start script to do this too.

Having installed the app this way, I have developed a process to update the code. I rename the previously demeteorized app directory, eg. to old; go into my Meteor app directory and type: demeteorize -o ../demeteoredapp, then mv ../old/.git ../demeteoredapp. At that point I can do git add --all and commit and push the new version up to my git repo on Webfaction. I then ssh onto the server and type:

cd webapps/nodeapp/demeteoredapp/appname
git pull; ../bin/stop ; ../bin/start

to pull in the new changes and launch them.

Note there is no need for the forever package when you do it this way. I’m not sure we need the demeteorizer package either, or if the Meteor bundler is sufficient.

I found some comments online that there can be memory problems if you use Meteor and MongoDB on Webfaction. I haven’t experienced this problem. (I’m keeping a close eye on it – I have even built an admin panel which includes a call to a Meteor method that executes the memory_usage.py script provided by Webfaction.)

Ongoing jobs

You will then want to set up cron jobs to start your MongoDB app if it fails for any reason. I have modelled this script off Webfaction’s start script for node.js:

#!/bin/sh
mkdir -p $HOME/webapps/mongoapp/run
mkdir -p $HOME/webapps/mongoapp/log
pid=$(/sbin/pidof $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod)
if echo "$pid" | grep -q " "; then
  pid=""
fi
if [ -n "$pid" ]; then
  user=$(ps -p $pid -o user | tail -n 1)
  if [ $user = "your-username" ]; then
    exit 0
  fi
fi
nohup $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod --auth --fork --logpath $HOME/webapps/mongoapp/log/mongo.log --dbpath $HOME/webapps/mongoapp/data/ --port 18000 > /dev/null 2>&1 &
/sbin/pidof $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod > $HOME/webapps/mongoapp/run/mongo.pid

Put this into your scheduled tasks using crontab -e.

You will also want to back up your database. A simple command to back up the database to a directory called dbbackup is:

$HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongodump --port 18000 -o $HOME/dbbackup

You will need to add the admin username and password.

I like to have daily, weekly and monthly backups, so I make three directories $HOME/mongo_backup/daily, weekly, monthly and have a script for each:

The daily script is:

#!/bin/sh
# $HOME/scripts/cron_daily_mongo_backup.sh
# add this to crontab, e.g. to run at 15:45 server time every day:
# 45 15 * * * ~/scripts/cron_daily_mongo_backup.sh

# back up the database
$HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongodump --port 18000 -o $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` 2>> $HOME/mongo_backup/daily/cron.log

# delete backup directories older than 7 days
find $HOME/mongo_backup/daily -type d -ctime +7 | xargs -r rm -rf  # Be careful using -rf

The weekly script is:

#!/bin/sh
# add this to crontab as, e.g. to run at 16:20 server time every Sunday:
# 20 16 * * 0 ~/scripts/cron_weekly_mongo_backup.sh
cp -R $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` $HOME/mongo_backup/weekly
find $HOME/mongo_backup/weekly -type d -ctime +32 | xargs -r rm -rf

And the monthly script is:

#!/bin/sh
# add this to crontab as, e.g. to run at 16:25 server time every 1st of the month:
# 25 16 1 * * ~/scripts/cron_monthly_mongo_backup.sh
#
cp -R $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` $HOME/mongo_backup/monthly
find $HOME/mongo_backup/monthly -type d -ctime +370 | xargs -r rm -rf

If you’ve read this far, please check out the finished product at signup.zone!

  

Serve datatables with ajax from Django

Datatables is an amazing resource which lets you quickly display lots of data in tables, with sorting, searching and pagination all built in.

The simplest way to use it is to populate the table when you load the page.  Then the sorting, searching and pagination all just happen by themselves.

If you have a lot of data, you can improve page load times by just serving the data you need to, using ajax. On first sight, this is made easy too.  However, be warned: if the server is sending only the data needed, then the server needs to take care of sorting, searching and pagination. You will also need to control the table column sizes more carefully.

There’s quite a lot required to get this right, so I thought I’d share what I’ve learned from doing this in Django.

Start with the following html. This example demonstrates using the render function to insert a link into the table.

</pre>
<div class="row">
<table class="table table-striped table-bordered" id="example" style="clear: both;">
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
</table>
</div>
<pre>

and javascript:

$(document).ready(function() {
    exampleTable = $('#example').dataTable( {
        "aaSorting": [[ 2, "asc" ]],
        "aoColumns": [
            { "mData":"name", "sWidth":"150px" },
            { "mData":"supplier", "sWidth":"150px",
              "mRender": function (supplier, type, full)  {
                             return '<a href="'+supplier.slug+'">' + supplier.name + '</a>';
                         },
            },
            { "sType": 'numeric', "sClass": "right", "mData":"price", "sWidth":"70px" },
        ],
        "bServerSide": true,
        "sAjaxSource": "{% url 'api' 'MyClass' %}",
        "bStateSave" : true, // optional
                fnStateSave :function(settings,data){
                        localStorage.setItem("exampleState", JSON.stringify(data));
                },
                fnStateLoad: function(settings) {
                        return JSON.parse(localStorage.getItem("exampleState"));
                },
        fnInitComplete: function() { // use this if you don't hardcode column widths
            this.fnAdjustColumnSizing();
        }
    });
    $('#example').click(function() { // only if you don't hardcode column widths
        exampleTable.fnAdjustColumnSizing();
    });

Next you need to write an API for the data. I’ve put my api in its own file, apis.py, and made it a generic class-based view, so I’ve added to urls.py:

from django.conf.urls import patterns, url
from myapp import views, apis

urlpatterns = patterns('',
   ...
   url(r'^api/v1/(?P<cls_name>[\w-]+)/$',apis.MyAPI.as_view(),name='api'),
)

Then in apis.py, I put the following. You could use Django REST framework or TastyPie for a fuller solution, but this is often sufficient. I’ve written it in a way that can work across many classes; just pass the class name in the URL (with the right capitalization). One missing feature here is an ability to sort on multiple columns.

import sys
import json

from django.http import HttpResponse
from django.views.generic import TemplateView
from django.core.serializers.json import DjangoJSONEncoder

import myapp.models

class JSONResponse(HttpResponse):
    """
    Return a JSON serialized HTTP response
    """
    def __init__(self, request, data, status=200):
        # pass DjangoJSONEncoder to handle Decimal fields
        json_data = json.dumps(data, cls=DjangoJSONEncoder)
        super(JSONResponse, self).__init__(
            content=json_data,
            content_type='application/json',
            status=status,
        )

class JSONViewMixin(object):
    """
    Return JSON data. Add to a class-based view.
    """
    def json_response(self, data, status=200):
        return JSONResponse(self.request, data, status=status)

# API

# define a map from json column name to model field name
# this would be better placed in the model
col_name_map = {'name': 'name',
                'supplier': 'supplier__name', # can do foreign key look ups
                'price': 'price',
               }
class MyAPI(JSONViewMixin, View):
    "Return the JSON representation of the objects"
    def get(self, request, *args, **kwargs):
        class_name = kwargs.get('cls_name')
        params = request.GET
        # make this api general enough to handle different classes
        klass = getattr(sys.modules['myapp.models'], class_name)

        # TODO: this only pays attention to the first sorting column
        sort_col_num = params.get('iSortCol_0', 0)
        # default to value column
        sort_col_name = params.get('mDataProp_{0}'.format(sort_col_num), 'value')
        search_text = params.get('sSearch', '').lower()
        sort_dir = params.get('sSortDir_0', 'asc')
        start_num = int(params.get('iDisplayStart', 0))
        num = int(params.get('iDisplayLength', 25))
        obj_list = klass.objects.all()
        sort_dir_prefix = (sort_dir=='desc' and '-' or '')
        if sort_col_name in col_name_map:
            sort_col = col_name_map[sort_col_name]
            obj_list = obj_list.order_by('{0}{1}'.format(sort_dir_prefix, sort_col))

        filtered_obj_list = obj_list
        if search_text:
            filtered_obj_list = obj_list.filter_on_search(search_text)

        d = {"iTotalRecords": obj_list.count(),                # num records before applying any filters
            "iTotalDisplayRecords": filtered_obj_list.count(), # num records after applying filters
            "sEcho":params.get('sEcho',1),                     # unaltered from query
            "aaData": [obj.as_dict() for obj in filtered_obj_list[start_num:(start_num+num)]] # the data
        }

        return self.json_response(d)

This API depends on the model for two extra things:

  • the object manager needs a filter_on_search method, and
  • the model needs an as_dict method.

The filter_on_search method is tricky to get right. You need to search with OR on the different fields of the model, and AND on different words in the search text. Here is an example which subclasses the QuerySet and object Manager classes to allow chaining of methods (along the lines of this StackOverflow answer).

from django.db import models
from django.db.models import Q
from django.db.models.query import QuerySet

class Supplier(models.Model):
    name = models.CharField(max_length=60)
    slug = models.SlugField(max_length=200)

class MyClass(models.Model):
    name = models.CharField(max_length=60)
    supplier = models.ForeignKey(Supplier)
    price = models.DecimalField(max_digits=8, decimal_places=2)
    objects = MyClassManager()

    def as_dict(self):
        """
        Create data for datatables ajax call.
        """
        return {'name': self.name,
                'supplier': {'name': self.supplier.name, 'slug': self.supplier.slug},
                'price': self.price,
                }

class MyClassMixin(object):
    """
    This will be subclassed by both the Object Manager and the QuerySet.
    By doing it this way, you can chain these functions, along with filter().
    (A simpler approach would define these in MyClassManager(models.Manager),
        but won't let you chain them, as the result of each is a QuerySet, not a Manager.)
    """
    def q_for_search_word(self, word):
        """
        Given a word from the search text, return the Q object which you can filter on,
        to show only objects containing this word.
        Extend this in subclasses to include class-specific fields, if needed.
        """
        return Q(name__icontains=word) | Q(supplier__name__icontains=word)

    def q_for_search(self, search):
        """
        Given the text from the search box, search on each word in this text.
        Return a Q object which you can filter on, to show only those objects with _all_ the words present.
        Do not expect to override/extend this in subclasses.
        """
        q = Q()
        if search:
            searches = search.split()
            for word in searches:
                q = q & self.q_for_search_word(word)
        return q

    def filter_on_search(self, search):
        """
        Return the objects containing the search terms.
        Do not expect to override/extend this in subclasses.
        """
        return self.filter(self.q_for_search(search))

class MyClassQuerySet(QuerySet, MyClassMixin):
    pass

class MyClassManager(models.Manager, MyClassMixin):
    def get_query_set(self):
        return MyClassQuerySet(self.model, using=self._db)

This is a stripped down version of my production code. I haven’t fully tested this stripped down version, so please let me know if you find any problems with it.

Hope it helps!

  

Better than jQuery.ajax() – Django with Angular

I have built a website for my games company Second Nature Games using Django.  Django brings lots of benefits like a nice admin panel where my partner can upload new game boards and add new content to the site.

However, to write games on the web you can’t rely too much on a server-side framework like Django. You are going to have to write some javascript as well.  I was keen for the challenge, having used some jQuery before.

In my first attempt, the boards were rendered by Django on the server, and then the javascript on the client-side would manipulate the document object model (DOM) as the game was played. As I described in an earlier post, I used jQuery’s $.ajax() command to alert the server when the game was finished, and then the javascript would update scores and stats as required.

But this is not easily scalable:  as I added a leaderboard and more stats, more and more things needed to be updated, and I got into a tangle.

The solution is to use a javascript MVC framework like Backbone, Angular or Ember (or Meteor even more so).  I chose to use Angular. There is a great website, To Do MVC, which shows the same To Do application written in many different frameworks, so you can make an informed choice if you’re facing the same decision.

Using one of these frameworks changes how you view the server: the server just sends data and the client renders it. This is known as “data on the wire”.  The effect is to turn the server into an API. (Unfortunately for Django fans, I suspect Django is really overkill for this. But having developed the rest of the site with Django, I stuck with it.)  Meanwhile, whenever the data updates, the javascript framework will update the DOM for you.

Here is a sample template for the Panguru leaderboard, using Angular:

<div class="info">
  <div class="leaderboard">
    <h3 class="title">Leaderboard</h3>
    <ul>
      <li ng-repeat="entry in leaderboard.entries">
        <span class="name">{{ entry.name }}</span>
        <span class="score">{{ entry.score }}</span>
      </li>
    </ul>
  </div>
</div>

You can see it’s pretty straightforward and elegant. The code to load it from the API is:

$scope.updateLeaderboard = function() {
  var that = this;
  $http.get('api/leaderboard/')
    .success(function(data, status, headers, config) {
      that.leaderboard.entries = data;
    })
    .error(function(data, status, headers, config) {
      console.log('error updating leaderboard');
    });
  }

As you may have noticed, Angular and Django both use the double-parentheses notation. You don’t want to have them both try to interpret the same file.

One approach is to use Django’s {% verbatim %} tag. I think it’s nicer to separate the Django-rendered templates from the Angular-rendered templates completely, into two separate directories. Django templates stay in their app’s templates directory. I put Angular’s html files into the app’s static/partials directory. If the server needs to provide data to the javascript, I let Django render it in its template, and pick it up in the static html file. One example is to pass the csrf token. Another example arises because I’m using the staticfiles app, so Django renames all the static files. Unfortunately this includes the locations of all the partial html files and images you need. E.g. in my Django templates/game_page.html:

<html lang="en" ng-app="panguru">
<head>
  <title>Panguru Online</title>
  ...
  {% addtoblock "css" %}{# if you are using sekizai #}</pre>
  <style type="text/css">
    .load {background-image: url('{% static "img/loader.gif" %}');}
  </style>
  {% endaddtoblock %}
  <script type="text/javascript">
    csrf_token = '{{ csrf_token }}';
    container_url = '{% static "partials/container.html" %}';
    ...
  </script>

</head>
<body>
  <div panguru-container></div>
</body>
</html>

This part I’m not especially proud of, and would love to hear if you have a better solution.

You can then either use a Django API framework like TastyPie to serve the api/leaderboard/ URL, or you can write one yourself. I did this starting from a JSON response mixin like the one in the Django docs, or this one by Oz Katz, developed for use with Backbone. Then, in Django views.py, I put:

class LeaderboardApi(JSONViewMixin, View):
    "Return the JSON representation of the leaderboard"
    def get(self, request, *args, **kwargs):
        return self.json_response(Score.as_list(limit=20))

That just requires a method on your model which returns the object in an appropriate format, e.g.

class Score(models.Model):
    user = models.ForeignKey(User)
    score = models.PositiveIntegerField(default=0)
    class Meta:
        ordering = ['-score']
    @classmethod
    def as_list(cls):
        return [score.as_dict() for score in cls.objects.all()]
    def as_dict(self):
        return {'name': self.user.username, 'score': self.score }

There’s just one more thing, which is an inconsistency in how Django and Angular apply trailing slashes to URLs. Unfortunately if you use Angular’s $resource approach to interacting with the API, it will strip off any final slash, and then Django will redirect the URL to one with a slash, losing any POST parameters along the way. There’s a fair bit of discussion about this online. My approach has been to just turn off Django’s default behavior in settings.py:

APPEND_SLASH = False

Get ready to thoroughly test your app when you make this change, especially if you are also using Django-CMS, as I am. I uncovered a few places where I had been a bit sloppy in hardwiring URLs, and was inconsistent in my use of slashes in them.

The full result is the logic puzzle game Panguru, which you can play online or buy as a physical boardgame.  Please check it out and let me know what you think!

Drag and drop with AngularJS and ng-repeat

I am using AngularJS and jQuery UI and have an app with drag-and-drop behaviour. I want to update a statistic on the screen every time an item is dropped.  The items are displayed in an  ng-repeat.

This turns out to be a bit tricky for a newcomer to AngularJS, e.g. you need to know:

I haven’t seen any examples that put all this together for my particular case, and though the solution is straightforward it took me quite a while to piece it together.

To save you the time, here is my solution.  To simplify it, it just updates a text string obj.outer. When you load up the page, the n and inner are replaced with the numbers 1,2,3 and “Inner linked”, as expected. outer is replaced with “Updated on link”, which is also what you would expect. You can drag and drop the pieces of text on each other. And when you do, the inner and outer fields are updated.

index.html:

    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
    <script src="//cdnjs.cloudflare.com/ajax/libs/angular.js/1.1.5/angular.min.js"></script>
    <script src="//cdnjs.cloudflare.com/ajax/libs/underscore.js/1.5.1/underscore-min.js"></script>
    <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js"></script>

    <div ng-app="test">
    <p ng-init="obj={outer:'not yet'}">outer: {{obj.outer}}</p>
      <ul>
          <li ng-repeat="n in [1,2,3]"><span my-dnd>{{n}} {{inner}}</span></li>
      </ul>
    </div>

app.js:

    angular.module('test', [])
      .directive('myDnd', function() {
        return {
            link: function(scope, elt, attrs) {
                scope.inner = "Inner";
                scope.obj.outer = "Updated on link";
                elt.draggable({revert:"invalid"});
                elt.droppable({
                    accept: function( d ) { return true; },
                    drop: function( event, ui ) {
                        return scope.$apply(function() {
                            scope.inner = 'Dropped';
                            scope.obj.outer = 'Updated on drop';
                        });
                    }
                });
              }
            }
        });

Some comments:

  • Note that you can use scope.obj.outer; you do not need to use scope.$parent.obj.outer. I believe this is ‘prototypical Javascript inheritance’. I’m not exactly sure why you need the $parent sometimes but not others.
  • You definitely need scope.obj.outer rather than just scope.outer.
  • You can leave out the return in the statement return scope.$apply(...). I haven’t looked into what the difference is. You cannot leave the call to $apply out.
  • I would like to know why the documentation shows the link function using scope rather than $scope, as used by the controllers.

Please let me know if you have any thoughts or improvements!

  

Private media with Django

I have often wanted user-uploaded files and images to be “private” or “secure”, i.e. require some authentication and authorisation to view, but haven’t known how to start. Now that I have a solution that works (e.g. for sites hosted by Webfaction), I’d like to share it with you.

The basic principles are:

  1. Serve your public “static” files (e.g. css and javascript) and any public user/admin-uploaded “media” files from your existing static webapp. This static webapp has no capability to selectively hide some files from view, so we will not use it for the private media.
  2. Serve your private media from a regular Django view:
    • This view will be accessed through a regular Django URL in a urls.py file.
    • The first step in this view is to authenticate and check for authorisation. Be aware that you will only have the request parameters (including the URL path) to do this with. One solution might be to use a directory structure with permissions varying by directory.
    • The second step is to serve the file. You could do this using Django directly, but it would be woeful. Webfaction uses Apache, which has a nice module called mod_xsendfile which lets Apache serve the contents (Django merely specifies the path in the response header). There is a solution if you use nginx too which I have not explored (see django-filer’s secure downloads feature for more). You need to explicitly install and activate this module however – see below for details.
  3. Store your private media somewhere different to your public media. You can do this by providing a custom storage for your private FileFields and ImageFields. If you need instance-specific permissions, you can do this by passing a method as the upload_to directory.

Note – before going down this path, check if the file system you are using already has a protection mechanism (e.g. Amazon’s S3 service), in which case you probably won’t need this.

An implementation

I packaged up an implementation of these principles in django-private-media. You can install it from PyPi with:

pip install django-private-media

This package draws significantly from the secure file download code of Stephan Foulis’s django-filer.  One key difference between the two is that django-filer replaces all file and image fields with a foreign key; in contrast, because my focus is solely on the permissioning, django-private-media just uses the standard Django file and image fields.  As a result, it should be fairly straightforward to convert an existing project to use it.  See the readme for more details.

Code snippets

You’ll find below some code snippets to point you in the right direction. They assume you add a PRIVATE_MEDIA_URL and PRIVATE_MEDIA_ROOT, analogous to Django’s MEDIA_URL and MEDIA_ROOT, to your settings.py file.

urls.py

# urls.py
from django.conf import settings
...
urlpatterns += patterns('appname.views',
    url(r'^{0}(?P.*)$'.format(settings.PRIVATE_MEDIA_URL.lstrip('/')), 'serve_private_file',),
)
...

views.py

# views.py
from django.conf import settings

def has_read_permission(self, request, path):
    "Only show to authenticated users - extend this as desired"
    # Note this could allow access to paths including ../..
    # Don't use this simple check in production!
    return request.user.is_authenticated()

def serve_private_file(request, path):
    "Simple example of a view to serve private files with xsendfile"
    if has_read_permission(request, path):
        fullpath = os.path.join(settings.PRIVATE_MEDIA_ROOT, path)
        response = HttpResponse()
        response['X-Sendfile'] = fullpath
        return response

models.py

# models.py

from django.db import models
from django.conf import settings

private_media = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT,
                                  base_url=settings.PRIVATE_MEDIA_URL,
                                  )
# or you could define a custom subclass of FileSystemStorage

class Car(models.Model):
    photo = models.ImageField(storage=private_media)

settings.py

# settings.py
PRIVATE_MEDIA_ROOT = '/home/username/private-media' # for example
PRIVATE_MEDIA_URL = '/private/' # for example

Installing xsendfile

To install and activate xsendfile on Webfaction, follow the advice given by this post.

That’s all!

What have I missed?  Please let me know if you’ve done something similar and have another or better solution.

Otherwise, I hope this helps!