Task queues

Task queues manage background work that must be executed outside the usual HTTP request-response cycle.

Why are task queues necessary?

Tasks are handled asynchronously either because they are not initiated by an HTTP request or because they are long-running jobs that would dramatically reduce the performance of an HTTP response.

For example, a web application could poll the GitHub API every 10 minutes to collect the names of the top 100 starred repositories. A task queue would handle invoking code to call the GitHub API, process the results and store them in a persistent database for later use.

Another example is when a database query would take too long during the HTTP request-response cycle. The query could be performed in the background on a fixed interval with the results stored in the database. When an HTTP request comes in that needs those results a query would simply fetch the precalculated result instead of re-executing the longer query. This precalculation scenario is a form of caching enabled by task queues.

Other types of jobs for task queues include

  • spreading out large numbers of independent database inserts over time instead of inserting everything at once

  • aggregating collected data values on a fixed interval, such as every 15 minutes

  • scheduling periodic jobs such as batch processes

Task queue projects

The defacto standard Python task queue is Celery. The other task queue projects that arise tend to come from the perspective that Celery is overly complicated for simple use cases. My recommendation is to put the effort into Celery's reasonable learning curve as it is worth the time it takes to understand how to use the project.

  • The Celery distributed task queue is the most commonly used Python library for handling asynchronous tasks and scheduling.

  • The RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. RQ is backed by Redis and is designed to have a low barrier to entry.

  • Taskmaster is a lightweight simple distributed queue for handling large volumes of one-off tasks.

  • Huey is a Redis-based task queue that aims to provide a simple, yet flexible framework for executing tasks. Huey supports task scheduling, crontab-like repeating tasks, result storage and automatic retry in the event of failure.

  • Kuyruk is simple and easy to use task queue system built on top of RabbitMQ. Although feature set is small, new features can be added by extensions.

  • Dramatiq is a fast and reliable alternative to Celery. It supports RabbitMQ and Redis as message brokers.

Hosted message and task queue services

Task queue third party services aim to solve the complexity issues that arise when scaling out a large deployment of distributed task queues.

  • Iron.io is a distributed messaging service platform that works with many types of task queues such as Celery. It also is built to work with other IaaS and PaaS environments such as Amazon Web Services and Heroku.

  • Amazon Simple Queue Service (SQS) is a set of five APIs for creating, sending, receiving, modifying and deleting messages.

  • CloudAMQP is at its core managed servers with RabbitMQ installed and configured. This service is an option if you are using RabbitMQ and do not want to maintain RabbitMQ installations on your own servers.

Open source examples that use task queues

Task queue resources

Task queue learning checklist

  1. Pick a slow function in your project that is called during an HTTP request.

  2. Determine if you can precompute the results on a fixed interval instead of during the HTTP request. If so, create a separate function you can call from elsewhere then store the precomputed value in the database.

  3. Read the Celery documentation and the links in the resources section below to understand how the project works.

  4. Install a message broker such as RabbitMQ or Redis and then add Celery to your project. Configure Celery to work with the installed message broker.

  5. Use Celery to invoke the function from step one on a regular basis.

  6. Have the HTTP request function use the precomputed value instead of the slow running code it originally relied upon.

What's next to learn after task queues?

How do I log errors that occur in my application?

I want to learn more about app users via web analytics.

What tools exist for monitoring a deployed web app?

Sign up for two emails per month with Python tutorials and Full Stack Python updates.

Full Stack Python

Full Stack Python is an open book that explains concepts in plain language and provides helpful resources for those topics.
Updates via newsletter, Twitter & Facebook.
1. IntroductionLearning ProgrammingWhy Use Python?Python 2 or 3?Enterprise PythonPython CommunityCompanies using PythonBest Python ResourcesBest Python VideosBest Python Podcasts2. Development EnvironmentsText Editors & IDEsVimEmacsSublime TextPyCharmJupyter NotebookBash shellShellsZshPowerShellTerminal MultiplexerstmuxScreenPymuxEnvironment configurationApplication DependenciesSource ControlGitMercurialApache SubversionHosted Source ControlGitHubBitBucketGitLab3. Core LanguageGeneratorsComprehensions4. TestingUnit TestingIntegration TestingCode MetricsDebuggingLoggingMarkdown6. Security7. Web DevelopmentWeb FrameworksDjangoFlaskBottlePyramidFalconMorepathSanicOther Web FrameworksTemplate EnginesJinja2MakoDjango TemplatesWeb DesignHTMLCascading Style Sheets (CSS)Responsive DesignMinificationCSS FrameworksBootstrapJavaScriptTask QueuesCeleryRedis Queue (RQ)DramatiqStatic Site GeneratorsPelicanLektorMkDocsWebSocketsuvloop8. DeploymentServersStatic ContentPlatform-as-a-ServiceVirtual Private ServersOperating SystemsUbuntuWeb ServersApache HTTP ServerNginxCaddyWSGI ServersGreen Unicorn (Gunicorn)Continuous IntegrationJenkinsConfiguration ManagementAnsibleDockerServerlessAWS LambdaGoogle Cloud Functions9. DataRelational DatabasesPostgreSQLMySQLSQLiteObject-relational MappersSQLAlchemyPeeweeDjango ORMSQLObjectPony ORMNoSQL Data StoresRedisMongoDBApache CassandraNeo4jData analysispandasBokehd3.jsMatplotlib10. Web APIsMicroservicesBotsAPI CreationAPI IntegrationTwilio11. DevOpsMonitoringCachingRollbarWeb Analytics12. Change LogWhat Full Stack MeansAbout the AuthorFuture DirectionsPage Statuses ...or view all topics.

Matt Makai 2012-2018