Using redis-queue for asynchronous calls with Django

I recently posted about using Redis and Celery with Django to handle asynchronous calls from your web pages. Given that I have memory constraints on the server, I have been wondering if I might get more bang for my buck with redis-queue (rq) instead of Celery.  In fact, I have found them comparable: rq uses about 12Mb per worker, and Celery uses about 10-12Mb per process.  However, Celery workers use (1+concurrency) processes, so if concurrency=1, Celery appears to use double the memory.

Using RQ

Here are the changes I’ve made to the code I posted earlier to replace Celery with redis-queue.  Note jobs.py is exactly the same as celery’s tasks.py, but without the @task decorator. (I did not use rq’s @job decorator.)

def status_view(request):
    """
    Called by the opt page via ajax to check if the optimisation is finished.
    If it is, return the results in JSON format.
    """
    if not request.is_ajax():
        raise SuspiciousOperation("No access.")
    if QUEUE_BACKEND=='celery':
        # as before - the main part was a call to Celery's AsyncResult
    elif QUEUE_BACKEND=='rq':
        from django.conf import settings
        from redis import Redis
        from rq import Queue
        from rq.job import Job, Status
        from rq.exceptions import NoSuchJobError
        try:
            connection = Redis(settings.RQ_REDIS_URL, settings.RQ_REDIS_PORT)
            # not quite sure if better to use Job(...) or Job.fetch(...) here
            # the difference is fetch also calls refresh
            # but I see it does not rerun the job
            job = Job.fetch(request.session['job_id'], connection=connection)
        except KeyError, NoSuchJobError:
            ret = {'error':'No optimisation is underway (or you may have disabled cookies).'}
            return HttpResponse(json.dumps(ret))
        if job.is_finished:
            ret = get_solution(job)
        elif job.is_queued:
            ret = {'status':'in-queue'}  # note extra
        elif job.is_started:
            ret = {'status':'waiting'}
        elif job.is_failed:
            ret = {'status': 'failed'}   # note extra

def get_context_data(self, **kwargs):
    ...
    if QUEUE_BACKEND=='celery':
        from . import tasks
        result = tasks.solve.delay(myarg, timeout=timeout)
    elif QUEUE_BACKEND=='rq':
        from . import jobs
        from redis import Redis
        from rq import Queue
        connection = Redis('localhost', PORT)
        q = Queue(connection=connection)
        job = q.enqueue_call(func=jobs.solve, args=[myarg],
                             kwargs={'timeout':timeout}, timeout=timeout+10)
        # the solve call itself has a timeout argument; timeout with rq shouldn't occur

In settings.py I added:

     RQ_REDIS_URL = 'localhost'
     RQ_REDIS_PORT = 6379

But I did not use django-rq at all.

One nice thing I see immediately is the additional status info – you can easily query if a job is still in the queue or has failed.  I’m sure these are possible to see in Celery too, but they are obvious in rq.

Run RQ workers

Running an rq worker is nice and simple – there is no daemonization or even setup files. On either your dev or production server, just type (and repeat for as many workers as you want):

rqworker --port 6379

Remaining issues

One initial problem was finding out how to get an existing job from its id.  I solved this with:

Job.fetch(job_id, connection=connection)

However, I cannot find documentation about Job.fetch, and I see that Job(...) by itself also works.  Please let me know if you know which of these I should be using.

The main problem I have with redis-queue now is terminating a task.  I have a “cancel” button on the optimisation screen, which I can implement with Celery via:

revoke(task_id, terminate=True)  # celery

I cannot find an equivalent in rq.  This is unfortunately a deal-breaker for me, so I am sticking with celery for now.  Can you help?

  

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>