Inside Class Cruncher

You’ll find here the concepts used to create Class Cruncher, a handy webapp for school administrators.

The problem

How should you deploy a certain number of teachers across your school’s classes, given rules about class sizes? You may need to use composite classes.

More precisely, given:

  • The total number of teachers
  • Which grades are being taught (eg. kindergarten, 1st grade, 2nd grade etc)
  • How many children are in each grade
  • Min & max class sizes in each grade
  • Which composite classes are allowed (eg. grades 1 & 2 but not grades 1 & 3 together)
  • The smallest number of children from a grade allowed in a composite class (eg. so you don’t have a single lonely first grader in a class with grade 2).

Then list out the classes that each teacher should take (i.e. how many children from each grade are in each class).

A solution

Solve this as a mixed-integer linear programming problem, using lpsolve. This uses the simplex and branch-and-bound algorithms to maximise an objective function subject to some constraints. Here we have lots of constraints, and just want a feasible solution, rather than an “optimal” solution, so the objective function is not so important.

Step 1: Choose your variables

For every potential class j, we’ll have a binary variable y(j) (ie. it must be either 0 or 1), to tell us if it is being used or not.

Because classes can include students from different grades, we will invent the term “subclass” to refer to the number of children from one particular grade in a class. Then our key variables are x(i), the number of children in each potential subclass i.

Let’s relate the classes and subclasses via a matrix C(j,i), with 1 if subclass i is in class j, and 0 otherwise. Since subclasses are only in a single class, for any subclass i, C(j,i) is 1 for only one class j. So C is mostly 0s, with each row having at least one 1, and no columns having more than one 1.

Then the number of children in class j = the sum over i of C(j,i).x(i).

(In matrix notation, Cx = number of children in each class, where x and the right hand side are column vectors.)

C(j,i) also tells us if subclass i is being used: it is given by the sum over j of C(j,i).y(j).

Recall that y(j) tells us if class j is being used. Also, for a given subclass i, C(j,i) is only 1 for a single class j, so the sum is just a single term.

(In matrix notation, C’y = binaries telling if each subclass is used, where C’ is the transpose of C.)

For example: Suppose you have 3 grades (kindergarten, 1 and 2), and can allow for 1 pure kindy class, 2 composite K/1 classes, 2 pure 1st grade and 1 pure 2nd grade class. That’s a total of 6 classes and 8 subclasses. The matrix C is:

—-8 subclasses—–
| 1 . . . . . . .
| . 1 1 . . . . .
6 . . . 1 1 . . .
classes . . . . . 1 . .
| . . . . . . 1 .
| . . . . . . . 1

To simplify the model, we’ll assume potential composite classes cannot optionally only use some of their subclasses – it’s all or nothing.

So in fact, we also need a preliminary step (before solving) to decide on the number of potential classes and subclasses from the input data. We can make a guess at an upper limit, eg. by dividing the number of children in each grade by the maximum number allowed in a class, and rounding up.

Step 2: Choose your constraints

Our constraints need to achieve these things:

  1. Ensure all the children in each grade are in a class (or equivalently, a subclass)
  2. Classes sizes must be between the minimum and maximum class size – or zero (since not all potential classes need to be used!)
  3. Subclass sizes must be above the minimum subclass size, if the class is being used
  4. The number of classes must match (or be less than) the number of available teachers

We also need to relate the x and y variables. In a linear program, all the constraints need to be linear in the variables z. This means you can express them as matrices A, so that A.z is greater than, equal to, or less than, another vector c. Here z is just a column vector containing both x and y.

Take each of the above in turn.

Ensure all the children in each grade are in a class (or equivalently, a subclass)

We have to express this in terms of the number of children in the subclasses x, because the y’s don’t tell us how many children there are in each class.

In our earlier example, the first point involves 3 constraints, one for each grade. Multiply the below matrix with the column vector x (which has 8 rows).

—-8 subclasses—– sums to
| 1 1 . 1 . . . . = number of kindergarteners
3 grades . . 1 . 1 1 1 . = number of first graders
| . . . . . . . 1 = number of 2nd graders

Classes sizes must be between the minimum and maximum class size, or zero

If we had to use every potential class (ie. all the y variables were known to be 1), then this is just the constraint Cx >= min_class_size, Cx <= max_class_size (where I’m applying the inequality to each row of each side, so these are 6+6=12 constraints in our example.)

However, Cx >= min_class_size should only apply if y is 1. That’s easy to write: Cx – min_class_size.y >= 0. So the constraint matrix has C (N_classes × N_subclasses or 6×8) on the left and -min_class_size (N_classes × N_classes or 6×6, a diagonal matrix of the minimum class sizes) on the right.

I make the same modification for the max class sizes, although it’s not strictly necessary; this has the effect of forcing y >= 0.

Subclass sizes must be above the minimum subclass size, if the class is being used

If we had to use every potential class (ie. all the y variables were known to be 1), and therefore every subclass, then this is a very simple set of constraints: x >= min_subclass_size. However, we only want this constraint to apply if the corresponding binary value is 1. That corresponding y is actually the corresponding row of C’y (as we discussed earlier).

There’s a classic trick to only apply constraints based on the value of a binary variable, called the “big M” method.

Write x >= min_subclass_size as (x – min_subclass_size) >= 0, and then replace the right hand side with M(C’y – 1), where M is a big number (eg. 10000). Now when C’y is 1, the constraint applies; when it is 0, it does not. Cool!

The number of classes must match (or be less than) the number of available teachers

The last constraint is easy – the sum of the y’s must be less than or equal the number of teachers; the x’s don’t come into it.

Do we need to explicitly constrain y to be binary?

Certainly all our variables need to be integers, not floating point. But y already can’t be bigger than 1, because our big M constraint would fail. And it can’t be less than 0 because of the way we wrote our max class size constraint.

So y is already forced to be binary.

Matrix representation of the constraints

A:  (integer vars)      (binary vars)                     b:
 -- num_subclasses -- -- num_classes --
(--------------------+-----------------)                  (--------------)
(                    |                 )      |           (       |      )
( grade_size_matrix  |        0        )  num_grades  =   (  grade_sizes )
(                    |                 )      |           (       |      )
(--------------------+-----------------)                  (--------------)
(                    |-l1              )      |           (       |      )
( class_size_matrix  |     (diag.)     )  num_classes >=  (       0      )
(                    |             -lnc)      |           (       |      )
(--------------------+-----------------)                  (--------------)
(                    |-u1              )      |           (       |      )
( class_size_matrix  |     (diag.)     )  num_classes <=  (       0      )
(                    |             -unc)      |           (       |      )
(--------------------+-----------------)                  (--------------)
( 1                  |                 )      |           (       |      )
(     1              |                 )      |           (       |      )
(        ...         |     -M * C'     ) num_sub_classes>=(    ls - M    )
(              1     |                 )      |           (       |      )
(                  1 |                 )      |           (       |      )
(--------------------+-----------------)                  (--------------)
(         0          | 1 1   ...   1 1 ) num_classes =,<= ( max_classes  )
(--------------------+-----------------)                  (--------------)

where l is the min_class_size
      u is the max_class_size
      ls is the min_subclass_size
      C' is the transpose of the class_size_matrix
      M is a large number

Step 3: Choose the objective function

For a single answer, I just minimise the sum of all the x(i).

Ideally you could get second-best and other feasible solutions from the solver, but I had trouble doing this. As a work-around, I produce different solutions by changing the weights on each subclass. I raise the weight from 1 to 2 for the subclasses in a given grade or composite class, and repeat.

Step 4: Express the solution in an understandable way

We recast the resulting x and y values so that people can understand them, eg. in our example with 8 subclasses we might get an answer like

--------------- x ----------- ------- y -------
[20, 12, 10, 0, 0, 25, 22, 21, 1, 1, 0, 1, 1, 1]

This is more easily understood as being 5 classes with the following compositions:

           K  1st 2nd
class #1: 20,  0,  0
class #2: 12, 10,  0
class #3:  0, 25,  0
class #4:  0, 22,  0
class #5:  0,  0, 21

Step 5: Post-process to humanise the answers

The optimisation produces often solutions with very different numbers of students in each class, eg. two kindergarten classes, one with 10 students, and one with 20. Clearly, a better solution is two classes of 15.

As is often the case, the constraints we thought we needed don't quite fully give us the solutions we expected.

We could tackle this by adding more constraints or changing the objective function, but in this case it is simpler to do some "shuffling" after the optimisation, according to some prescribed rules.

Composite classes make this a bit harder. For another example, given the formatted output (note the composite classes are listed at the end):

[[10,0,0], [20,0,0], [0,25,0], [0,0,10], [25,5,0], [0,5,25]]

We could return:

[[21,0,0], [21,0,0], [0,22,0], [0,0,20], [13,8,0], [0,5,15]]

The rules I came up with took some discovering:

  1. For each grade, find the average class size of classes with subclasses from that grade. (In the above example, it's 20, which includes 5 grade 1s in the comp class)
  2. For each grade, move that grade's students around to make them as close to this average as possible, eg:
    • kindergarten: avg = 20;

      [[20,0,0], [20,0,0], [0,25,0], [0,0,10], [15,5,0], [0,5,25]]

    • grade 1: avg = (25+20+30)/3 = 25; in the example, this cannot be done, since the last class is the problem.
    • grade 2: avg = (10+30)/2 = 20,

      [[20,0,0], [20,0,0], [0,25,0], [0,0,20], [15,5,0], [0,5,15]]

  3. Repeat until class sizes only change by 1, eg:
    • kindergarten: avg = 20, no shuffling required
    • grade 1: avg = (25+20+20)/3 = 21.7;

      [[20,0,0], [20,0,0], [0,22,0], [0,0,20], [15,7,0], [0,6,15]]

    • grade 2: avg = (20+21)/2 = 20.5, no change required
    • kindergarten: avg = (20+20+22)/3 = 20.7,

      [[21,0,0], [20,0,0], [0,22,0], [0,0,20], [14,7,0], [0,6,15]]

    • etc

Step 6: Build it!

Build a web app to take the inputs, perform the optimisation and serve the results!

  

Dropdown menus in Meteor

I’ve been developing a site in Meteor, using their default templating engine Blaze.

And here’s a question: what’s the Meteor-like way to show a dropdown menu?

I’m using Bootstrap, so of course we can use Bootstrap’s javascript (based on jQuery) to do this. But it doesn’t feel quite right – shouldn’t we use a reactive approach, eg. with a reactive Session variable telling us whether the dropdown is open?

Surprisingly, I haven’t seen any such examples in my Google travels.

So until I get my act together to publish this as a Meteor package, here is a way to do it.

Define this html template:

<template name="dropdown">
    {{> Template.dynamic template=templateName data=templateData}}
</template>

And put this javascript in your client folder. Note it uses underscore for _.extend, but you should be able to remove this dependency if you need to.

//
// if the ESC key is pressed or a mouse is clicked anywhere, close any open dropdowns
//

$(document).keyup(function(evt) {
    if (evt.keyCode === 27) {
        Session.set('dropdown', null);
    }
});

// You will need to change the word "layout" here to be a template
// that you define which covers your entire webpage
Template.layout.events({
    'click': function() {
        Session.set('dropdown', null);
    }
});

Template.dropdown.helpers({

    "templateName": function() {
        return 'dropdown_' + this.name;
    },

    "templateData": function() {
        // add an 'open' property to the template for the child to tell if it is open
        // _.extend(dest, src) copies all the properties in src to dest and returns dest
        return _.extend({open: Session.equals('dropdown', this.name)}, this);
    }

});

Template.dropdown.events({

    'click button': function(evt) {
        evt.preventDefault();
        evt.stopPropagation();  // stops the full-page click handler above from firing
        Session.set('dropdown', this.name);
    },

    'click .dropdown-menu li a': function(evt) {
        evt.preventDefault();
        Session.set('dropdown', null);
    }

});

Note that I’ve defined a click handler which fires on any click anywhere on the webpage (you need to change the line Template.layout.events to your own page-wide template), and which closes any open dropdowns. I then use evt.stopPropagation() to stop that firing when you click the button which opens the dropdown (otherwise it could never open). If you define any other click events which should not affect open dropdowns, then you need to include evt.stopPropagation() in their definition.

With that done, it’s easy to put a dropdown in your template. Just include:

{{>dropdown name='example'}}

This finds the template dropdown_example, which will have access to an “open” method to tell if it is open. So you can define it like this:

<template name="dropdown_example">
    {{#if open}}
        <ul class="dropdown-menu x">
            <li><a href="#">Option 1</a></li>
            <li><a href="#">Option 2</a></li>
            <li><a href="#">Option 3</a></li>
        </ul>
        <button class="invisible btn">Try this</button>
    {{else}}
        <button class="btn">Try this</button>
    {{/if}}
</template>

If you need to pass the dropdown menu template some data, you can pass it directly:

{{>dropdown name='example' someData='foo'}}

and then dropdown_example will have access to the variable someData.

Hope that helps! If you have a better solution, or find any mistakes, please let me know.

  

Testing React components with Jest

Jest makes a bold claim to be “Painless Javascript Unit Testing”. I’m looking forward to using it, but it hasn’t quite lived up to that claim for me yet. Hopefully this post will help it do so for you.

If you just want a finished example, check out the tests in react-dropdown-input.

Choose the right version of node and jest

On my Mac, the latest version of jest (0.4.0) doesn’t run with the latest version of node (0.12.0). With node 0.10.37, it only works intermittently – sometimes saying:

dyld: lazy symbol binding failed: Symbol not found: _node_module_register
Expected in: dynamic lookup

and sometimes saying:

Error: Worker process exited before responding! exit code: null, exit signal: SIGSEGV (or SIGTRAP)
A worker process has quit unexpectedly! This is bad news, shutting down now!

The solution is to use jest version 0.2.1 with node 0.10.x. Hopefully future versions of jest will overcome this.

Note that the latest version of Babel seems to require node 0.12.x.

jest.dontMock is not recursive

I am testing a component called DropdownInput, built on top of React-Bootstrap, eg for DropdownInput.js:

    var ReactBootstrap = require('react-bootstrap');
    var Input = ReactBootstrap.Input;
    var DropdownMenu = ReactBootstrap.DropdownMenu;
    // in render function: 
      return (
        <div>
          <Input .... />
          <DropdownMenu 
            className={this.props.menuClassName}
            ....>
          </DropdownMenu>
        </div>);

I wrote a test like this (in __tests__.DropdownInput-test.js):

    jest.dontMock('../DropdownInput');
    var React = require('react/addons');
    var DropdownInput = require('../DropdownInput');
    var TestUtils = React.addons.TestUtils;
    
    describe('DropdownInput', function() {
      it('contains the test class', function() {
        var elt = (<DropdownInput menuClassName='test'/>);
        var renderedItem = TestUtils.renderIntoDocument(elt);
        TestUtils.findRenderedDOMComponentWithClass(
                                renderedItem, 'test');
      });
    });

This could not find any DOM elements with the class test:

Error: Did not find exactly one match (found: 0) for class:test

The reason is that while jest.dontMock('../DropdownInput') uses the real DropdownInput, it does not use the real ReactBootstrap components. Which is fair enough. So you need to add:

jest.dontMock('react-bootstrap');

But that’s not enough! The reason is that ReactBootstrap contains the line require('classnames'), which help it put class names on its components; but jest is still mocking classnames. So you need to also add:

jest.dontMock('classnames');

Happily, that does the job. It feels fragile though, because I don’t care what packages ReactBootstrap uses, as long as it works. If the author of ReactBootstrap changes which packages it uses, I don’t want all my tests breaking! Is there a better way?

There is a discussion of the pros and cons of jest’s approach to mocking here.

Shallow and deep rendering

There’s a good argument that your component tests should not try to test the subcomponent behavior anyway. In that case, you might be happy with a ‘shallow’ test like this:

    // __tests__/DropdownInput-shallow-test.js
    //
    // Check it has the right components
    //

    jest.dontMock('../DropdownInput');
    var React = require('react/addons');
    var DropdownInput = require('../DropdownInput');
    var ReactBootstrap = require('react-bootstrap');
    var TestUtils = React.addons.TestUtils;
    
    describe('DropdownInput', function() {
      var menuClassName = 'test';
      var elt = (<DropdownInput menuClassName={menuClassName}/>);
      var result;

      beforeEach(function() {
        var shallowRenderer = TestUtils.createRenderer();
        shallowRenderer.render(elt);
        result = shallowRenderer.getRenderOutput();
      });

      it('contains the right components', function() {
        // for now, assume in that order - TODO: generalize
        var child0 = result.props.children[0];
        var child1 = result.props.children[1];
        expect(child0.type).toEqual(ReactBootstrap.Input);
        expect(child1.type).toEqual(ReactBootstrap.DropdownMenu);
        // and check the menu is passed the right class name
        expect(child1.props.className).toEqual(menuClassName);
      });

      it('contains the right first menu item', function() {
        // again, not a very general approach
        var child1 = result.props.children[1];
        var child1props = child1.props.children[0];
        expect(child1props[0].props.children).toContain(names[0]);
      });

    });

And this works nicely. However, as it is written, it assumes a lot about the structure of the component, like what order the elements are in. If I decide to wrap my component in an extra div for some reason, it will break my tests. Is there an equivalent to TestUtils.findRenderedDOMComponentWithType that we can use for shallow rendering, so I don’t have to assume ordering?

But you could also argue the opposite case: that the tests shouldn’t care how DropdownInput produces the desired behavior, only that it does – ie. it should test the behavior, not which components it uses. For this you need a ‘deep’ test, with lots of the mocking turned off. Eg:

    // __tests__/DropdownInput-deep-test.js
    //
    // Check it has the right behavior
    //
    jest.dontMock('../DropdownInput');
    jest.dontMock('react-bootstrap');
    jest.dontMock('classnames');
    var React = require('react/addons');
    var DropdownInput = require('../DropdownInput');
    var ReactBootstrap = require('react-bootstrap');
    var TestUtils = React.addons.TestUtils;

    describe('DropdownInput', function() {
      var menuClassName = 'test';
      var elt = (<DropdownInput menuClassName={menuClassName}/>);
      var renderedItem;

      beforeEach(function() {
        renderedItem = TestUtils.renderIntoDocument(elt);
      });

      it('has a working input', function() {
        var txt = 'a';
        var input = TestUtils.findRenderedDOMComponentWithTag(
                        renderedItem, 'input').getDOMNode();
        //TestUtils.Simulate.keyDown(input, {key: 'a'});  // this doesn't work
        TestUtils.Simulate.change(input,  {target: {value: txt}});
        expect(input.value).toEqual(txt);
      });

    });

And this works nicely too, except that the test needs to know which packages to unmock. (And why doesn’t Simulate.keyDown work?)

So, at this early stage in my jest career, I plan to include both sorts of tests for each component, in separate files.

Flux components

If you’re using the Flux architecture, then user interaction will often fire an action, rather than directly changing the DOM. So instead of testing the DOM directly, we need to test that an action was sent. To do this, I adapted the tutorial on Testing Flux Applications (which describes testing stores), as follows:

    var element,
        cpt,
        rendered,
        ActionCreators,
        MyComponent;

    describe('MyComponent', function() {

        beforeEach(function() {
          // by requiring a new copy for every test, 
          // we reset the mock call counters for each test
          MyComponent = require('../my-component.js');
          ActionCreators = require('../../../actions/action-creators');
          // Render it
          element = <MyComponent />;  // include relevant props
          cpt = TestUtils.renderIntoDocument(element);
          rendered = TestUtils.findRenderedDOMComponentWithTag(cpt, 'div');
          // note - this version doesn't work - why??
          // rendered = TestUtils.findRenderedComponentWithType(cpt, MyComponent);
        });

        it('clicking on it sends the right action', function() {
          // first check that no action calls were made yet
          // ActionCreators.myAction is the desired action
          var actionCalls = ActionCreators.myAction.mock.calls;
          expect(actionCalls.length).toEqual(0);
          // Simulate a click and verify that it sends out the action
          TestUtils.Simulate.click(rendered);
          expect(actionCalls.length).toEqual(1);
          // check it sent the right params (replace ... with args)
          expect(ViewActionCreators.myAction).toBeCalledWith(...);
        });
    });

(You won’t find this in the DropdownInput component tests, since it doesn’t fire any actions.)

Conclusion

I hope this saves you some time and development pain. Please let me know if you have better solutions to any of these issues!

The working DropdownInput component and tests are available at github.com/RacingTadpole/react-dropdown-input; you can see also a demo here.

  

React Dropdown Input

React-dropdown-input is a small React component I’ve written that is available as an npm package. It lets you choose from a dropdown menu by typing into an input box.

If the example below doesn’t work, you can open it in its own page.

I’ve also written up a short post on how I’ve approached writing unit tests for it.

Please let me know if you have any comments!

  

Flux & React Best Practices

So, you want to start a new javascript project using React, perhaps using the Flux architecture. Excellent! But you’re not quite sure where to begin. Well, actually, you know where to begin, but you’ve quickly hit a barrage of design questions. Should that API call go in the store, the view component, or the action creator? Should I store that data in a state or pass it down from a parent component through props? How should I update the view before I receive that ajax data? And so on.

Here are some helpful tips which can guide you along the way. I am no authority on React, so I have included references to those who are. Please leave a comment if you think there are better ways.

Components & State

Try to keep as many of your components as possible stateless.

A common pattern is to create several stateless components that just render data, and have a stateful component above them in the hierarchy that passes its state to its children via props.

State should contain data that a component’s event handlers may change to trigger a UI update.

https://facebook.github.io/react/docs/interactivity-and-dynamic-uis.html

Ajax APIs

In Flux, all ajax API calls should come from the action creators.

paraphrased from http://facebook.github.io/flux/docs/chat.html

In response, the API fires actions on success (eg. “receive data”) or failure (eg. “receive error”).

For example, ViewActionCreators.js:

// ViewActionCreators.js
var AppAPI = require('../apis/AppAPI');
var ViewActionCreators = {
  doSomething: function(something) {
    AppDispatcher.handleAction({
      actionType: AppConstants.DO_SOMETHING,
      something: something
    });
    // fire off API call
    AppAPI.fetchData(something);
  }
}

AppAPI.js:

// AppAPI.js
// this example uses jQuery's ajax call
var ServerActionCreators = require('../actions/ServerActionCreators');
function post(url) {
  $.ajax({url: url
    success: function(data) {
      ServerActionCreators.receiveData(data);
    },
    error: function(xhr, status, err) {
      ServerActionCreators.receiveAPIError({ err: err });
    }
  }
}

ServerActionCreators.js:

// ServerActionCreators.js
var ServerActionCreators = {
  receiveData: function(data) {
    AppDispatcher.handleAction({
      actionType: AppConstants.RECEIVE_DATA,
      data: data
    });
  },
  receiveAPIError: function(err) {
    AppDispatcher.handleAction({
      actionType: AppConstants.RECEIVE_API_ERROR,
      err: err
    });
  }
}

In this code, the user interacts with the view, which fires the DO_SOMETHING action. That action itself fires off the API call, and when the API call is done it sends either a RECEIVE_DATA or a RECEIVE_API_ERROR action, which the Store can listen for. (As we’ll see below, the Store may listen for DO_SOMETHING too.)

These snippets are based on the Flux demo Chat app.

It seems there is one exception to the rule that only Action Creators call the API: loading initial data. This can go directly in the view controller. See https://facebook.github.io/react/tips/initial-ajax.html for an example.

Stores

Stores should remain as independent and decoupled as possible — a self-contained universe that one can query from a Controller-View.
The only road into the Store is through the callback it registers with the Dispatcher. The only road out is through getter functions.
Stores also publish an event when their state has changed, so Controller-Views can know when to query for the new state, using the getters.

http://stackoverflow.com/questions/23591325/in-flux-architecture-how-do-you-manage-store-lifecycle

Stores should only execute synchronous code. Otherwise they are too hard to understand.

http://www.code-experience.com/async-requests-with-react-js-and-flux-revisited/

Immutability

If you’re using Flux, you should start writing your stores using immutable-js.

If we also use immutable-js data structures to hold the components’ state, we could mix PureRenderMixin into all our components and short circuit the re-rendering process.

https://facebook.github.io/react/docs/advanced-performance.html#immutable-js-and-flux

Here is how I’ve done it. In the Store, make changes like this:

var Immutable = require('immutable');
var selected = Immutable.List();  // was: var selected = [];
// ...
selected = selected.push(item);  // was: selected.push(item);
// ...
selected = selected.delete(i);   // was: selected.splice(i, 1);
// ...
// I receive a js array of js objects from the API
// and convert to immutable list of immutable maps: 
  var tmp = myArray.map(function(item) {
    return Immutable.Map(item);
  });
  return Immutable.List(tmp);
// ...

In the Component, there are not many changes required – I don’t even require('immutable') in this example. The changes arise just because the objects returned by the Store are no longer plain old javascript objects, but immutable ones.

var React = require('react/addons');
var PureRenderMixin = React.addons.PureRenderMixin;

// no change needed state-setting function if it's like:
function getMyState() {
  return {
    a: MyStore.getSelected(),
    b: MyStore.getOther()
  };
}

React.createClass({
  mixins: [PureRenderMixin],      // add this
  // no change to these, eg.
  getInitialState:function(){
    return getMyState();
  },
  _onChange: function() {  // ChangeListener callback
    this.setState(getMyState());
  },
  // ...
var num = this.state.a.count();  // was this.state.a.length 
// maybe other small changes due to API differences

Here is an example of the ToDo app rewritten with Flux and immutablejs.

The React documentation also refers to another immutability helper, update(). This uses ordinary Javascript objects, whereas immutablejs uses its own object types. I have decided to use immutablejs, but please tell me if you have a way to do it with update.

Optimistic Updates

While waiting for an ajax API to return, you often want to optimistically update the UI. I’ve settled on the following pattern to do this:

  • Optimistically update the Store. Do not bypass the store by only saving it in the component state.
  • This means you will update the same Store data twice: once based on the user action, and then again based on the received API action (or API error action).
  • If you need to know if the API action has completed, you can set another variable (eg. optimistic) to track that, along with a getter function that the component can listen for. I’m using this to display a spinner while some data loads.

I tried doing each of these on its own too, but then you either can’t easily update the view optimistically, or you get a messy chain of actions.

The potential downsides are:

  • There could be other reasons for the API actions being received
  • I’m not sure I want this store to know about the API-generated actions (though the alternative still needs the API error action)
  • What if an API call is not required, and the data comes from somewhere else?

Reacting to non-React events

Attach generic DOM events (eg. window resizing) to componentDidMount. This is well explained at http://facebook.github.io/react/tips/dom-event-listeners.html

In the window resizing example, note they do not attempt to keep the current window size in a Store. Instead, it is kept in the component’s state. It would be complicated to keep a Store synched up with window.innerWidth, which is anyway the ultimate truth here.

Testing components

It took me a while to get Jest tests working, due to a combination of version conflicts, trouble understanding the nature of auto-mocking, and then deciding which things to mock. I’ve written up a separate post on unit testing components.

Other resources

Here are some resources (well, one anyway) that helped me get going:

  

Django & Angular overview

Angular is what HTML would have been if it had been designed for building web applications”

What problem does Angular solve?

It separates your javascript models, views and controllers – just like Django does for your server-side code.

It does so using “two-way data-binding”: whenever the model changes, the view changes as well – and vice versa.

Pros and Cons of Angular

Angular has a rich ecosystem of modules, eg. Ionic for mobile app development.

However, Angular 2 (to be released in 2015) will not be easily backwards compatible. Angular 1 may not be supported for much longer (18 months?).

Plenty of alternatives exist – check them out at ToDo MVC.

One that is gaining popularity is React – “a javascript library for building user interfaces”. Mark Finger has written a helpful package called django-react to make this easy to use in Django.

A quick Angular demo

Eg. see the code snippets on the Angular home page.

What tools make it easier to use with Django?

Server-side:

  • Django-angular – lots of useful utilities to help the two work together, especially around forms and template sharing; there is also support for ‘three-way’ data-binding (ie. the server detects when the client’s model changes – and the server can modify values on the client side without the client needing to poll).
  • Django REST framework or TastyPie – since your Django app’s API is now its main feature
  • Django-compressor or django-pipeline – because you will have dozens of little js files defining your Angular components

Client-side:

  • Grunt or gulp – to automate javascript necessities like minification, compilation, unit testing, linting, etc
  • Npm or bower – like pip install for your javascript packages
  • Angular has lots of modules you can add, eg. ngDialog and AngularUI
  • Don’t use the default angular router; ui-router is better.

And Yeoman – a “generator ecosystem” – although there is no django + angular generator yet.

What practices make it easier to use with Django?

This section derived from the excellent Thinkster tutorial Build Web Applications with Django and AngularJS.

Angular directory structure (in the project directory root):

  • /static/javascripts/<ng_app_name>.config.js
  • /static/javascripts/<ng_app_name>.js
  • /static/javascripts/<ng_app_name>.routes.js
  • /static/javascripts/<ng_module_name>/<ng_module_name>.module.js
  • /static/javascripts/<ng_module_name>/controllers/<controller_name>.controller.js, …
  • /static/javascripts/<ng_module_name>/directives/<directive_name>.directive.js, …
  • /static/javascripts/<ng_module_name>/services/<service_name>.service.js, …
  • /static/templates/<ng_module_name>/<ng_template_name>.html, …
  • /templates/<django_template_name>.html, …
  • /templates/javascripts.html

urls.py

urlpatterns = patterns(
    '',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^api/v1/', include(router.urls)),
    # pass everything else through to Angular
    url('^.*$', IndexView.as_view(), name='index'),
)

views.py

from django.views.decorators.csrf import ensure_csrf_cookie
from django.views.generic.base import TemplateView
from django.utils.decorators import method_decorator

class IndexView(TemplateView):
    template_name = 'index.html'

    @method_decorator(ensure_csrf_cookie)
    def dispatch(self, *args, **kwargs):
       return super(IndexView,self).dispatch(*args,**kwargs)

Testing frameworks

There are many javascript testing frameworks available, eg. mocha and jasmine.

What problems have people had?

Please let me know!

Resources – Tutorials

What is this post anyway?

These are some questions for and notes from the SyDjango meetup on Angular in January 2015.

  

Curve fitting with javascript & d3

If javascript is up to amazing animations and visualizations with d3, maybe it’s up to non-linear curve fitting too, right?

Something like this, perhaps:



Here’s how I did it:

  • The hard work is done using Cobyla (“Constrained Optimization BY Linear Approximation”), which Anders Gustafsson ported to Java and Reinhard Oldenburg ported to Javascript, as per this stackoverflow post.
    Cobyla minimises a non-linear objective function subject to constraints.
  • The demo and its components (including cobyla) use the module pattern, which has the advantage of keeping the global namespace uncluttered.
  • To adapt Cobyla to this curve fitting problem, I wrote a short wrapper which is added onto the cobyla module as cobyla.nlFit(data, fitFn, start, min, max, constraints, solverParams). This function minimises the sum of squared differences (y1^2-y2^2) between the data points, (x,y1), and the fitted points, (x,y2).
  • The Weibull cumulative distribution function (CDF), inverse CDF and mean are defined in the “distribution” module. Thus distribution.weibull([2,1,5]) .inverseCdf(0.5) gives the median (50th percentile) of a Weibull distribution with shape parameter 2, scale parameter 1 and location parameter 5.
  • The chart is built with d3. I am building an open-source library of a few useful chart types, d3elements, which are added onto the d3 module as d3.elts. This one is called d3.elts.xyChart.
  • So the user interface doesn’t get jammed, I use a javascript web worker to calculate the curve fit. I was surprised how easy it was to set this up.
  • I apologise in advance that this sample code is quite complicated. If you see ways to make it simpler, please let me know.
  • Finally, this may be obvious, but I like the rigour that a tool like jshint imposes. Hence the odd comment at the top of fitdemo.js, /* global fitdemo: true, jQuery, d3, _, distribution, cobyla */

Check out the source code on bitbucket here.

Please let me know what you think!

  

D3 Time Series Chart with Zoom & Notes

This is an example of a reusable chart built using d3. The range (zoom) slider and the notes panel are also built in d3, as separate widgets, so they can be customized further.

Check the sample source code on bitbucket for the full description of how to use it; here is the essence (without notes):

    var rangeWidget = d3.elts.startEndSlider().minRange(365*24*3600*1000);
    var tsChart = d3.elts.timeSeriesChart().rangeWidget(rangeWidget);
    d3.csv('data.csv', function(data) {
        tsData = _.map(data, function(d) {return [d.date, d.price]});
        d3.select("#chart").datum(tsData).call(tsChart);
    }

To add notes to this, use:

    var rangeWidget = d3.elts.startEndSlider().minRange(365*24*3600*1000);
    var clickPanel = d3.elts.makeClickPanel();
    var tsChart = d3.elts.timeSeriesChart().rangeWidget(rangeWidget);
    tsChart.notesMarkerClick(function(elt, note, closer) {
        clickPanel(elt, note && ("<h3>"+note.title+"</h3><p>"+note.desc+"</p>")), closer);
    });
    d3.csv('data.csv', function(data) {
        d3.csv('wheatNotes.csv', function(notes) {
            tsChart.notes(notes);
            tsData = _.map(data, function(d) {return [d.date, d.price]});
            d3.select("#chart").datum(tsData).call(tsChart);
        }
    }

Hope you find it useful!

  

Visualising Flows in a D3 Chord Diagram with Hover

This is an example of a reusable chart built using d3.

The idea is that you have a matrix of the flows between one category (here, optimist/neutral/pessimist) to another (introvert/extrovert). `d3.elts.flowChord()` then converts this matrix into a chord diagram, with the option of hover text.

Check the sample source code on bitbucket for the full description of how to use it; here is the essence:

  var colors = d3.scale.ordinal().range(["#AAA", "steelblue", "green", "orange", "brown"]);
  var hoverHtml = {'Introvert': '<h1>Introverts</h1>Like to be by themselves', 
      'Extrovert': '<h1>Extroverts</h1>Like the company of other people', 
      'Optimist': '<h1>Optimists</h1>Look on the bright side of life',
      'Neutral': '<h1>Neutrals</h1>Life could be good, it could be bad',
      'Pessimist': '<h1>Pessimists</h1>See the glass half empty'}
  var chordDiagram = d3.elts.flowChord().colors(colors).hoverHtml(hoverHtml).rimWidth(30);
  var data = [['Disposition','Optimist','Neutral','Pessimist'],
              ['Introvert', 0.8, 0.4, 0.67], 
              ['Extrovert', 0.2, 0.6, 0.33]]
  d3.select("#flow").datum(data).call(chordDiagram);
  

D3 bar chart with zoom & hover

This is an example of a reusable chart built using d3. The range (zoom) slider is built in d3 too, as a separate widget, so it can be customized.

Check the sample source code on bitbucket for the full description of how to use it; here is the essence:

    var rangeWidget = d3.elts.startEndSlider().minRange(30);
    var myChart = d3.elts.barChart().rangeWidget(rangeWidget);
    myChart.mouseOver(function(el, d) { showHover(el, d) });
    myChart.mouseOut(function(el, d) { hideHover(el, d) });
    d3.csv('data.csv', function(data) {
        data = _.map(data, function(d) {return [d.id,d.price]});
        d3.select("#chart").datum(data).call(myChart);
    }
  

Mathematical web apps & visualisation