Ants on a Log

Using Celery to Efficiently Process External API Calls in Django

Scott Vitale, Founder @

APIs: Legos for Developers

  • Authentication
  • Social network engagement
  • Push notifications
  • Analytics
  • ...pretty much everything

No Pain, No Gain

Relying on an external API means suffering when that API experiences...

  • Outages
  • Slow response times
  • Maintenance windows
  • DDoS and other attacks

Imagine a world where

You run a cutting edge pet shop called

Kick Ass Pets
You've built a sweet Django app that interfaces with the TurtleAPI, a service that provides interesting facts on all varieties of shelled reptiles.
TurtleAPI lives up to its name. API calls can take seconds to execute.

Like any good manager, we need to delegate

Goals

  • Fast page loads to the user
  • Slow operations moved to background task
  • Notification / Push when complete

Enter Celery

  • Asynchronous Task Queue
  • Uses a broker for message passing (RabbitMQ, Redis)
  • Delegates tasks to one or more worker processes

Synchronous API Call

sequenceDiagram User->>+Django: Request page Django->>+API: API Request Note over API: 10s elapses API->>-Django: API Response Django->>-User: Page loads

Move API call to Worker

sequenceDiagram User->>+Django: Request page Django->>-User: Page skeleton Worker->>+API: API Request Note over API: 10s elapses API->>-Worker: API Response

Add Celery as Queue

sequenceDiagram User->>+Django: Request page Django-->>Celery: Work item Django->>-User: Page skeleton Celery-->>+Worker: Work item Worker->>+API: API Request Note over API: 10s elapses API->>-Worker: API Response Worker->>-Celery: Work result

Retrieve Results

sequenceDiagram User->>+Django: Request page Django-->>Celery: Work item Django->>-User: Page skeleton Celery-->>+Worker: Work item Worker->>+API: API Request Note over API: 10s elapses API->>-Worker: API Response Worker->>-Celery: Work result User->>+Django: Is it done? Django->>+Celery: Is it done? Celery->>-Django: Work result Django->>-User: Work result

Celery Implementation

  • Install Broker (not shown)
  • Install/Configure Celery
  • Define TurtleAPI call as a task
  • Add URL & JS poller to query task status
  • Start Celery worker

Install/Configure Celery


pip install celery
                
app / settings.py
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672'
CELERY_RESULT_BACKEND = CELERY_BROKER_URL
                
app / celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'server.settings')

from django.conf import settings

celery_app = Celery('kickasspets')

# Using a string here means the worker don't have to serialize the
# configuration object to child processes.
celery_app.config_from_object('django.conf:settings')

# Load task modules from all registered Django app configs.
celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
                
app / __init__.py
from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import celery_app

__all__ = ['celery_app']

Define TurtleAPI call as a task

Before: 
app / views.py
import requests

def get_turtle_descriptions():
    # Get turtle descriptions from the API, deserialize, and return as
    # python list
    api_response = requests.get(
        'http://localhost:8001/turtle-api/descriptions')
    return api_response.json()['turtles']

class TurtleView(View):
    def get(self, request):
        descriptions = get_turtle_descriptions()

        return render(request, 'turtles.html', {
            'page': 'turtles',
            'descriptions': descriptions
        })
After (task definition): 
app / tasks.py
from __future__ import absolute_import
from celery import shared_task
import requests

@shared_task
def get_turtle_descriptions():
    # Get turtle descriptions from the API, deserialize, and return as
    # python list
    api_response = requests.get(
        'http://localhost:8001/turtle-api/descriptions')
    return api_response.json()['turtles']
After (view): 
app / views.py
from __future__ import absolute_import
from . import tasks

class TurtleView(View):
    def get(self, request):
        task = tasks.get_turtle_descriptions.delay()

        return render(request, 'turtles.html', {
            'page': 'turtles',
            'task_id': task.id
        })

Add status endpoint

Add URL to urlpatterns (/status/<task_id>) 
app / urls.py
url(r'^status/(?P<task_id>>[a-f\d\-]+)', StatusView.as_view())
                
app / views.py
from __future__ import absolute_import
from celery.result import AsyncResult

class StatusView(View):
    def get(self, request, task_id):

        # Get task by ID and its status
        task_result = AsyncResult(task_id)
        task_is_complete = task_result.ready()

        if task_is_complete:
            # Retrieve response data
            data = task_result.get()
        else:
            data = None

        return JsonResponse({
            'complete': task_is_complete,
            'data': data
        })

Add Javascript poller

Start Celery worker


                celery -A app worker -l info

Bask in your improved user experience

Thank You

http://ants.vitale.io (presentation) https://github.com/svvitale/kick-ass-pets (demo)