Tag Archives: javascript

Make an animated & reusable barchart

Do you need a dynamic bar chart like this in your web page, with positive and negative values, and categories along the x-axis?

Dog breeds are on my mind at the moment as we just bought a new Sheltie puppy – this chart might show a person’s scores for each breed. Click the button above to see the next person’s (random) set of scores.

This is an example of a reusable chart built using d3. Using it is fairly simple, eg.:

<script src="http://d3js.org/d3.v3.min.js"></script>
<script src="barChart.js"></script>
<script>
    var points = [['Beagle',-10],
                  ['Terrier',30],
                  ['Sheltie',55],
                  ['Greyhound',-24]];
    var myChart = d3.elts.barChart().width(300);
    d3.select("body").datum(points).call(myChart);
</script>

Please check out the source code for barChart.js on bitbucket.

You may also find this helpful even if you don’t need a barchart, but want to understand how to build a reusable chart. I was inspired when I read Mike Bostock’s exposition, Towards reusable charts, but I found it took some work to get my head around how to do it for real – so I hope this example may help others too.

The key tricks were:

  • How to adjust Mike Bostock’s sample code to produce bars instead of a line and area
  • Where the enter(), update and exit() fit in (answer: they are internal to the reusable function)
  • How to call it, including whether to use data vs datum (answer: see code snippet above)
  • How to refer to the x and y coordinates inside the function (answer: the function maps them to d[0] and d[1] regardless of how they started out)

You can find a much fancier version of this chart, with a range slider and hover text, in this post.

Good luck!

  

How is your tax money being spent?

Want to know how your tax money is being spent? The Australian Budget Explorer is an interactive way to explore spending by portfolio, agency, program or even in more detail; compare 2014 against previous years; and search for the terms that interest you.

Australian Budget Explorer

This was produced in collaboration with BudgetAus. This year for the first time, the team at data.gov.au provided the Budget expenditure data in a single spreadsheet, which Rosie Williams (InfoAus) manipulated to include further data on the Social Services portfolio. The collaboration is producing lots of good visualisations, collected at AusViz.

I won’t editorialise about the Budget here; instead here is my data and extensions wishlist:

Look-through to include State budgets

The biggest line item (component) in the Federal Budget is $54 billion for “Administered expenses: Special appropriation GST Revenue Entitlements – Federal Financial Relations Act 2009″, which I take it is revenue to the States. I would love to be able to “look through” this item into how the States spend it.

The BudgetAus team has provided some promising data leads here.

Unique identifiers to track spending over time

One of the most frequent requests I get is to track changes in spending over time.

Unfortunately this is hard, as there are no unique identifiers for a given portfolio, program, agency or component. That means if the name changes from one year to the next, it is hard to work out which old name corresponds to which new name. E.g. In 2014, the Department of Employment & Workplace Relations has been split into the Department of Employment and the Department of Education, while the Environment portfolio used to be “Sustainability, Environment, Water, Population and Communities”.

It would be great to give all spending an identifier, and have a record of how identifiers map from one year to the next.

What money is actually spent?

How does the budget relate to what is spent? There is some info here at BudgetAus, but the upshot is “This might be a good task for a future group of volunteers”…

Revenue

There is revenue data available here – I haven’t looked at it carefully yet, but I hope to include it, if possible.

Cross-country comparison

It would be great to compare the percentages spent in key areas by governments across the world.
Maybe it’s already being done? To do this I’d need some standard hierarchy of categories (health, education, defence, and subdivisions of these, etc), and we’d need every country’s government (and every State government) to tag their spending by those categories. Sounds simple in concept but I bet it would be hard to make it happen.

In the meantime, my plan is to check quandl for data and see how far I can go with what’s there…

D3

Finally, many thanks to the authors for the awesome d3 package!

Conclusion

If you have any comments or know how to solve any of the data issues raised above, please let me know.

  

Serve datatables with ajax from Django

Datatables is an amazing resource which lets you quickly display lots of data in tables, with sorting, searching and pagination all built in.

The simplest way to use it is to populate the table when you load the page.  Then the sorting, searching and pagination all just happen by themselves.

If you have a lot of data, you can improve page load times by just serving the data you need to, using ajax. On first sight, this is made easy too.  However, be warned: if the server is sending only the data needed, then the server needs to take care of sorting, searching and pagination. You will also need to control the table column sizes more carefully.

There’s quite a lot required to get this right, so I thought I’d share what I’ve learned from doing this in Django.

Start with the following html. This example demonstrates using the render function to insert a link into the table.

</pre>
<div class="row">
<table class="table table-striped table-bordered" id="example" style="clear: both;">
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
</table>
</div>
<pre>

and javascript:

$(document).ready(function() {
    exampleTable = $('#example').dataTable( {
        "aaSorting": [[ 2, "asc" ]],
        "aoColumns": [
            { "mData":"name", "sWidth":"150px" },
            { "mData":"supplier", "sWidth":"150px",
              "mRender": function (supplier, type, full)  {
                             return '<a href="'+supplier.slug+'">' + supplier.name + '</a>';
                         },
            },
            { "sType": 'numeric', "sClass": "right", "mData":"price", "sWidth":"70px" },
        ],
        "bServerSide": true,
        "sAjaxSource": "{% url 'api' 'MyClass' %}",
        "bStateSave" : true, // optional
                fnStateSave :function(settings,data){
                        localStorage.setItem("exampleState", JSON.stringify(data));
                },
                fnStateLoad: function(settings) {
                        return JSON.parse(localStorage.getItem("exampleState"));
                },
        fnInitComplete: function() { // use this if you don't hardcode column widths
            this.fnAdjustColumnSizing();
        }
    });
    $('#example').click(function() { // only if you don't hardcode column widths
        exampleTable.fnAdjustColumnSizing();
    });

Next you need to write an API for the data. I’ve put my api in its own file, apis.py, and made it a generic class-based view, so I’ve added to urls.py:

from django.conf.urls import patterns, url
from myapp import views, apis

urlpatterns = patterns('',
   ...
   url(r'^api/v1/(?P<cls_name>[\w-]+)/$',apis.MyAPI.as_view(),name='api'),
)

Then in apis.py, I put the following. You could use Django REST framework or TastyPie for a fuller solution, but this is often sufficient. I’ve written it in a way that can work across many classes; just pass the class name in the URL (with the right capitalization). One missing feature here is an ability to sort on multiple columns.

import sys
import json

from django.http import HttpResponse
from django.views.generic import TemplateView
from django.core.serializers.json import DjangoJSONEncoder

import myapp.models

class JSONResponse(HttpResponse):
    """
    Return a JSON serialized HTTP response
    """
    def __init__(self, request, data, status=200):
        # pass DjangoJSONEncoder to handle Decimal fields
        json_data = json.dumps(data, cls=DjangoJSONEncoder)
        super(JSONResponse, self).__init__(
            content=json_data,
            content_type='application/json',
            status=status,
        )

class JSONViewMixin(object):
    """
    Return JSON data. Add to a class-based view.
    """
    def json_response(self, data, status=200):
        return JSONResponse(self.request, data, status=status)

# API

# define a map from json column name to model field name
# this would be better placed in the model
col_name_map = {'name': 'name',
                'supplier': 'supplier__name', # can do foreign key look ups
                'price': 'price',
               }
class MyAPI(JSONViewMixin, View):
    "Return the JSON representation of the objects"
    def get(self, request, *args, **kwargs):
        class_name = kwargs.get('cls_name')
        params = request.GET
        # make this api general enough to handle different classes
        klass = getattr(sys.modules['myapp.models'], class_name)

        # TODO: this only pays attention to the first sorting column
        sort_col_num = params.get('iSortCol_0', 0)
        # default to value column
        sort_col_name = params.get('mDataProp_{0}'.format(sort_col_num), 'value')
        search_text = params.get('sSearch', '').lower()
        sort_dir = params.get('sSortDir_0', 'asc')
        start_num = int(params.get('iDisplayStart', 0))
        num = int(params.get('iDisplayLength', 25))
        obj_list = klass.objects.all()
        sort_dir_prefix = (sort_dir=='desc' and '-' or '')
        if sort_col_name in col_name_map:
            sort_col = col_name_map[sort_col_name]
            obj_list = obj_list.order_by('{0}{1}'.format(sort_dir_prefix, sort_col))

        filtered_obj_list = obj_list
        if search_text:
            filtered_obj_list = obj_list.filter_on_search(search_text)

        d = {"iTotalRecords": obj_list.count(),                # num records before applying any filters
            "iTotalDisplayRecords": filtered_obj_list.count(), # num records after applying filters
            "sEcho":params.get('sEcho',1),                     # unaltered from query
            "aaData": [obj.as_dict() for obj in filtered_obj_list[start_num:(start_num+num)]] # the data
        }

        return self.json_response(d)

This API depends on the model for two extra things:

  • the object manager needs a filter_on_search method, and
  • the model needs an as_dict method.

The filter_on_search method is tricky to get right. You need to search with OR on the different fields of the model, and AND on different words in the search text. Here is an example which subclasses the QuerySet and object Manager classes to allow chaining of methods (along the lines of this StackOverflow answer).

from django.db import models
from django.db.models import Q
from django.db.models.query import QuerySet

class Supplier(models.Model):
    name = models.CharField(max_length=60)
    slug = models.SlugField(max_length=200)

class MyClass(models.Model):
    name = models.CharField(max_length=60)
    supplier = models.ForeignKey(Supplier)
    price = models.DecimalField(max_digits=8, decimal_places=2)
    objects = MyClassManager()

    def as_dict(self):
        """
        Create data for datatables ajax call.
        """
        return {'name': self.name,
                'supplier': {'name': self.supplier.name, 'slug': self.supplier.slug},
                'price': self.price,
                }

class MyClassMixin(object):
    """
    This will be subclassed by both the Object Manager and the QuerySet.
    By doing it this way, you can chain these functions, along with filter().
    (A simpler approach would define these in MyClassManager(models.Manager),
        but won't let you chain them, as the result of each is a QuerySet, not a Manager.)
    """
    def q_for_search_word(self, word):
        """
        Given a word from the search text, return the Q object which you can filter on,
        to show only objects containing this word.
        Extend this in subclasses to include class-specific fields, if needed.
        """
        return Q(name__icontains=word) | Q(supplier__name__icontains=word)

    def q_for_search(self, search):
        """
        Given the text from the search box, search on each word in this text.
        Return a Q object which you can filter on, to show only those objects with _all_ the words present.
        Do not expect to override/extend this in subclasses.
        """
        q = Q()
        if search:
            searches = search.split()
            for word in searches:
                q = q & self.q_for_search_word(word)
        return q

    def filter_on_search(self, search):
        """
        Return the objects containing the search terms.
        Do not expect to override/extend this in subclasses.
        """
        return self.filter(self.q_for_search(search))

class MyClassQuerySet(QuerySet, MyClassMixin):
    pass

class MyClassManager(models.Manager, MyClassMixin):
    def get_query_set(self):
        return MyClassQuerySet(self.model, using=self._db)

This is a stripped down version of my production code. I haven’t fully tested this stripped down version, so please let me know if you find any problems with it.

Hope it helps!

  

Dynamic web pages with Django, Ajax and jQuery

How do you update a page’s content on the fly?

Of course, the simple answer is to use ajax.

But there are many ways to implement ajax, and sometimes it isn’t necessary. My aim here is to describe some solutions, and when each is appropriate.

Specifically, I’ll deal with sites using jQuery, and written with Django (and Django-CMS in particular, but that’s not critical to this discussion). I will also show how I hooked up Twitter Bootstrap‘s modal windows to do what I needed.

These are the approaches I have found for dynamically updating your webpage. They all have their uses:

  • Toggle hidden content
  • Generate content on the client side
  • jQuery’s $.ajax()
  • jQuery’s $.load()
  • Dajax/Dajaxice
  • a javascript MVC framework like Angular

Researching this took me a few days and quite a bit of trial and error, so with any luck by posting I will save someone else that time.

Toggle hidden content

First, you can often do without ajax, by toggling hidden content (with either explicit javascript or a suitable Twitter Bootstrap component). Here’s a simple example:

html:

<input type="submit" class="btn-link" name="comment"
    value="Add a comment" id="toggle-content" />
<div id="extra-content">{% include "myform.html" %}</div>

javascript:

$(document).ready(function() {
    $("#extra-content").hide();
    $("#toggle-content").click(function(){
        $("#extra-content").show();
        $("#toggle-content").hide();
    });
});

This is fine if you know the potential extra content in advance, the server doesn’t need to know about it, and there’s not too much of it. It has the very strong advantage of being very fast (once the original page has loaded).

Generate content on the client side

To be honest, after implementing what seemed to be a wonderful solution with ajax, using the .load() method below, I am now rewriting my webpage to generate the content as needed in javascript. The ajax solution was just too slow, and as luck would have it, the content I need to show can be based on existing content. In fact, I hope it can be as simple as copying an element if I change the css to depend on the enclosing class.

jQuery’s $.ajax()

For some things, like telling the server that the user has finished playing a game, I am using jQuery’s $.ajax() javascript command. This combines easily with a Django view to send and receive data.

Your template will look something like this (thanks to this stack overflow post for explaining a simple way to pass the CSRF token):

html:

{% csrf_token %}
<div id="game-board" data-board-id="{{ board.pk }}" data-done-ref="{% page_url 'game-handler' %}game-over/">...</div>

javascript:

function gameOver() {
    var board = $('#game-board').attr('data-board-id');
    $.ajax({
        type: "POST",
        url: $('#game-board').attr('data-done-ref'),  // or just url: "/my-url/path/"
        data: {
            csrfmiddlewaretoken: document.getElementsByName('csrfmiddlewaretoken')[0].value,
            board: board,
            move_list: move_list.join(','),
        },
        success: function(data) {
            alert("Congratulations! You scored: "+data);
        },
        error: function(xhr, textStatus, errorThrown) {
            alert("Please report this error: "+errorThrown+xhr.status+xhr.responseText);
        }
    });
}

And your view will look something like this (you need to set up urls.py to point to this*):

view.py:

from django.http import HttpResponseRedirect, HttpResponse
from django.http import Http404

def game_over(request):
    if request.is_ajax():
        try:
            board_pk = int(request.POST['board'])
            moves = list(map(int, request.POST['move_list'].split(',')))
        except KeyError:
            return HttpResponse('Error') # incorrect post
        # do stuff, e.g. calculate a score
        return HttpResponse(str(score))
    else:
        raise Http404

(*since I am using Django-CMS, I have actually set this up as an apphook. To do this, the admin needs to manually connect a particular page to the apphook; I also require this page to have a nominated id in the advanced settings, e.g. “game-handler”, so that I can use {% page_url "game-handler" %} in the template. I have passed this from the template to the javascript so that the javascript can reside in a static js file.)

Here I am just passing a few POST parameters to the Django view, and the Django view processes them and returns the score – very simple handling of data. Json encoding of data is also fairly easy, I believe (e.g. see the Dajaxice example below).

The downside of this and the other ajax approaches is speed: any time you interact with the server, there is the potential for it to be slow. You may want to display a loading indicator along the lines of this stack overflow post.

jQuery’s $.load()

jQuery’s $.load() javascript command is magic! It makes ajax so easy you don’t even realise you’re doing it. I am using this to replace the contents of a div with new data from the server when a button is clicked.

The html and javascript is straightforward (this example is slightly more complex than it needs to be, but is my real-world situation; it is based on this stack overflow post for reloading the content of a Twitter Bootstrap modal popup window):

html:

<a href="my-ajax-page/load/{{ board.pk }}/" data-target="#my-modal">
   Click here
</a>

javascript:

$("a[data-target=#my-modal]").click(function(event) {
    event.preventDefault();
    var target = $(this).attr("href");
    $("#my-modal .modal-body").load(target, function() {
         $("#my-modal").modal("show");
    });
});

With the setup above, the contents of the view referenced by my-ajax-page/load/### will be loaded into the modal dialog and displayed.

In fact using the href, data-target and data-toggle tags in a Bootstrap modal window automatically calls jQuery’s .load command, but it only seems to do it the first time; in my case, I need to load new content every time the modal is clicked, hence the explicit call above.

A few caveats I have discovered about changing contents (but double-check me on this please!) -

  • Sekizai – I use sekizai blocks for my js and css (as done by Django-CMS). If you are inserting new css and js inside the html, you may want to do it without putting them inside sekizai blocks.
  • jQuery’s $(function() {…}) – if you insert new javascript code using this function (or the equivalent .ready() method), which on a normally loaded page is automatically run when the page is ready, it will not be run.

Dajax/Dajaxice

Dajax turns the paradigm on its head, allowing you to seemingly dynamically change your page’s content from python, rather than in the javascript. I haven’t looked into it, but I presume that under the hood the python code is sending an object to the javascript, and the Dajax js library is decoding it there.

This is slightly more involved to set up, requiring several changes to your Django setup (settings.py, templates etc), and I found the documentation a little sparse.  It also took me some time to get the basic examples running; I wasn’t clear on whether I needed Dajaxice or Dajax, or which was the one to get started with; I installed Dajax first not realising I had to also follow the Dajaxice instructions (which was my bad); then I had to play with the ordering of the INSTALLED_APPS to make it work (in between django.contrib.sites and django.contrib.messages).

Once set up though, your javascript is confined only to what happens when the ajax call returns; you can just use html to call the ajax script.  And your Django code is called without need to set your own urls up; Dajax autodiscovers them so long as you put your ajax views into files named ajax.py instead of view.py.

From the examples on the Dajax website, it looks like with Dajaxice you need to write your own javascript to handle the returned data, whereas with Dajax you can change page elements directly from python without writing any javascript at all. The Dajax API also allows you to assign attributes to elements, add or remove css classes, and other related functions. This stuff is not too hard to do in javascript or jQuery either though.

The key advantage of Dajax that I see is it helps maintain the separation of models and views, since the code which changes the page is in python rather than in the template.

Dajaxice

A Dajaxice example from their website is:

html:

<input type="text" id="text"/>
<input type="button"
    onclick="Dajaxice.examples.args_example(
    callback, {'text':$('#text').val()})"
    value="Send!"/>

javascript:

function callback(data){
    alert(data.message);
}

With the ajax.py file (in a Django app named “examples”, as referenced by the “onclick” function above):

ajax.py:

from django.utils import simplejson
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def args_example(request, text):
    return simplejson.dumps({'message':'Message is %s!' % text})

Dajax

And a Dajax example from the website, with one minor modification to make it clear that you do not need to write any javascript:

html:

<input type="text" value="5" id="a"> x
<input type="text" value="6" id="b"> =
<input type="text" value="" id="result">
<input type="button" value="Multiply!"
    onclick="Dajaxice.examples.multiply(
    Dajax.process,{'a':$('#a').val(),'b':$('#b').val()});">

With the ajax.py file (in a Django app named examples, as referenced by the onclick method above):

ajax.py:

from dajax.core import Dajax
from dajaxice.decorators import dajaxice_register

@dajaxice_register
def multiply(request, a, b):
    dajax = Dajax()
    result = int(a) * int(b)
    dajax.assign('#result','value',str(result))
    return dajax.json()

A javascript MVC framework like Angular

For web apps of any complexity, it is worth going with an MVC framework so you don’t have to manage the DOM updates yourself after the ajax call.  See this post for details of how I am getting Django to work with Angular.

Summary

All these approaches have their place. If you know the potential extra content in advance and the server doesn’t need to know about it, you can toggle hidden content. If you can generate it on the client side, that will probably be more responsive than using ajax. If you want to tell the server something but don’t need to change the content, you can use .ajax(). If you need to load in a whole new section, you can use .load() (but be aware it could be slow communicating with the server).

I think Dajaxice’s functionality is covered by .ajax(), but Dajaxice nicely handles the CSRF token for you and looks clearer. Dajax would be handy if several elements need to be changed and lets you write the page-changing code in python instead of javascript. It should thereby help maintain a cleaner separation of models and views.

I’m just learning this myself, so if I’ve missed another approach or said something misleading, please let me know!