Celery

Celery is a task queue implementation for Python web applications used to asynchronously execute work outside the HTTP request-response cycle.

Celery task queue project logo.

Celery is an implementation of the task queue concept. Learn more in the web development chapter or view the table of contents for all topics.

Why is Celery useful?

Task queues and the Celery implementation in particular are one of the trickier parts of a Python web application stack to understand.

If you are a junior developer it can be unclear why moving work outside the HTTP request-response cycle is important. In short, you want your WSGI server to respond to incoming requests as quickly as possible because each request ties up a worker process until the response is finished. Moving work off those workers by spinning up asynchronous jobs as tasks in a queue is a straightforward way to improve WSGI server response times.

What's the difference between Celeryd and Celerybeat?

Celery can be used to run batch jobs in the background on a regular schedule. A key concept in Celery is the difference between the Celery daemon (celeryd), which executes tasks, Celerybeat, which is a scheduler. Think of Celeryd as a tunnel-vision set of one or more workers that handle whatever tasks you put in front of them. Each worker will perform a task and when the task is completed will pick up the next one. The cycle will repeat continously, only waiting idly when there are no more tasks to put in front of them.

Celerybeat on the other hand is like a boss who keeps track of when tasks should be executed. Your application can tell Celerybeat to execute a task at time intervals, such as every 5 seconds or once a week. Celerybeat can also be instructed to run tasks on a specific date or time, such as 5:03pm every Sunday. When the interval or specific time is hit, Celerybeat will hand the job over to Celeryd to execute on the next available worker.

Celery tutorials

Celery is a powerful tool that can be difficult to wrap your mind around at first. Be sure to read up on task queue concepts then dive into these specific Celery tutorials.

Celery deployment resources

Do you want to learn more about task queues, or another topic?

How do I execute code outside the HTTP request-response cycle?

I've built a Python web app, now how do I deploy it?

What tools exist for monitoring a deployed web app?

Sign up for two emails per month with Python tutorials and Full Stack Python updates.

Sponsored By

Rollbar logo

Fix errors in your Python code before your users see them by monitoring with Rollbar.


Mapbox logo.

Easily build maps, search and navigation into your Python applications with Mapbox.


Real Python logo

Upgrade your Python skills by reading Real Python's awesome programming email newsletter.

Full Stack Python

Full Stack Python is an open book that explains concepts in plain language and provides helpful resources for those topics.
Updates via newsletter, Twitter & Facebook.
1. IntroductionLearning ProgrammingCore LanguageWhy Use Python?Python 2 or 3?Enterprise PythonPython CommunityCompanies using PythonBest Python ResourcesBest Python VideosBest Python Podcasts2. Development EnvironmentsText Editors & IDEsVimEmacsSublime TextPyCharmJupyter NotebookShellsBash shellZshPowerShellTerminal MultiplexerstmuxScreenPymuxEnvironment configurationApplication DependenciesVirtualenvsEnvironment variablesLocalhost tunnelsSource ControlGitMercurialApache SubversionHosted Source ControlGitHubBitBucketGitLab3. DataRelational DatabasesPostgreSQLMySQLSQLiteObject-relational MappersSQLAlchemyPeeweeDjango ORMSQLObjectPony ORMNoSQL Data StoresRedisMongoDBApache CassandraNeo4jData analysispandasNumPySciPyBokehd3.jsMatplotlibMarkdown4. Web DevelopmentWeb FrameworksDjangoFlaskBottlePyramidFalconMorepathSanicOther Web FrameworksTemplate EnginesJinja2MakoDjango TemplatesWeb DesignHTMLCascading Style Sheets (CSS)Responsive DesignMinificationCSS FrameworksBootstrapFoundationJavaScriptTask QueuesCeleryRedis Queue (RQ)DramatiqStatic Site GeneratorsPelicanLektorMkDocsTestingUnit TestingIntegration TestingCode MetricsDebuggingWebSocketsuvloopWeb APIsMicroservicesBotsAPI CreationAPI IntegrationTwilioSecurity5. DeploymentServersStatic ContentVirtual Private ServersPlatform-as-a-ServiceOperating SystemsUbuntuWeb ServersApache HTTP ServerNginxCaddyWSGI ServersGreen Unicorn (Gunicorn)Continuous IntegrationJenkinsConfiguration ManagementAnsibleDockerServerlessAWS LambdaGoogle Cloud Functions6. DevOpsMonitoringRollbarCachingLoggingWeb AnalyticsChange LogWhat Full Stack MeansAbout the AuthorFuture DirectionsPage Statuses ...or view all topics.

Matt Makai 2012-2018