Celery

Celery is a task queue implementation for Python web applications used to asynchronously execute work outside the HTTP request-response cycle.

Celery task queue project logo.

Celery is an implementation of the task queue concept. Learn more in the web development chapter or view the table of contents for all topics.

Why is Celery useful?

Task queues and the Celery implementation in particular are one of the trickier parts of a Python web application stack to understand.

If you are a junior developer it can be unclear why moving work outside the HTTP request-response cycle is important. In short, you want your WSGI server to respond to incoming requests as quickly as possible because each request ties up a worker process until the response is finished. Moving work off those workers by spinning up asynchronous jobs as tasks in a queue is a straightforward way to improve WSGI server response times.

What's the difference between Celeryd and Celerybeat?

Celery can be used to run batch jobs in the background on a regular schedule. A key concept in Celery is the difference between the Celery daemon (celeryd), which executes tasks, Celerybeat, which is a scheduler. Think of Celeryd as a tunnel-vision set of one or more workers that handle whatever tasks you put in front of them. Each worker will perform a task and when the task is completed will pick up the next one. The cycle will repeat continously, only waiting idly when there are no more tasks to put in front of them.

Celerybeat on the other hand is like a boss who keeps track of when tasks should be executed. Your application can tell Celerybeat to execute a task at time intervals, such as every 5 seconds or once a week. Celerybeat can also be instructed to run tasks on a specific date or time, such as 5:03pm every Sunday. When the interval or specific time is hit, Celerybeat will hand the job over to Celeryd to execute on the next available worker.

Celery tutorials

Celery is a powerful tool that can be difficult to wrap your mind around at first. Be sure to read up on task queue concepts then dive into these specific Celery tutorials.

Celery deployment resources

Do you want to learn more about task queues, or another topic?

How do I execute code outside the HTTP request-response cycle?

I've built a Python web app, now how do I deploy it?

What tools exist for monitoring a deployed web app?

Sign up here to receive a monthly email with major updates to this site, tutorials and discount codes for Python books.


Matt Makai 2012-2017