Baseline Celery 5 + Redis Setup for Offloading Slow API Work

A slow third-party call, a PDF render, an email batch — put any of these on the request path and your API’s tail latency belongs to someone else’s infrastructure. The fix is to stop doing the work inside the request and hand it to a worker instead.

This is a from-scratch async-jobs baseline: Celery 5 with Redis 7 as both broker and result backend, a FastAPI route that enqueues work and returns 202 Accepted with a status URL, and a worker that runs the same way on a laptop and on a production box. Nothing here is clever. It is the boring, correct setup that most teams reach eventually, written down so you can reach it on day one.

What actually belongs off the request path

The test is simple: if the caller does not need the result in the same round trip, the work does not belong in the handler. Generating a report, calling a slow upstream provider, resizing an upload, sending a batch of notifications — the client wants an acknowledgement that the work is accepted, not to hold a socket open while it runs. Keeping that work in the request handler ties your API’s latency, and its failure modes, to whatever the slow thing is having a bad day about.

Celery gives you a task queue: the API publishes a message to a broker, a separate worker process consumes it, and the result — success, failure, or return value — lands in a backend the API can poll. Redis plays both roles here. It is not the most durable broker in existence (RabbitMQ has stronger delivery semantics), but for a single-region service moving jobs measured in thousands per minute, Redis 7 is more than enough and one fewer moving part to operate.

The Celery app and the task

One module defines the Celery app, its configuration, and the tasks. Keep it importable by both the worker and the web process so there is a single source of truth for task signatures.

# app/worker.py
import os
import time

from celery import Celery

REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")

celery_app = Celery(
    "async_jobs",
    broker=REDIS_URL,
    backend=REDIS_URL,
)

celery_app.conf.update(
    # Serialise as JSON; never accept pickle over the wire.
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],
    timezone="UTC",
    enable_utc=True,

    # Reliability: only acknowledge a message once the task finishes,
    # and pull one task at a time so slow jobs are not hoarded.
    task_acks_late=True,
    worker_prefetch_multiplier=1,
    task_reject_on_worker_lost=True,

    # Bound how long a task may run. Soft raises SoftTimeLimitExceeded
    # so you can clean up; hard kills the worker child process.
    task_soft_time_limit=240,
    task_time_limit=300,

    # Let the status endpoint distinguish "queued" from "running".
    task_track_started=True,

    # Redis broker redelivers unacknowledged messages after this many
    # seconds. Keep it comfortably above your hard time limit.
    broker_transport_options={"visibility_timeout": 600},

    # Silence the 5.3+ startup-retry deprecation warning.
    broker_connection_retry_on_startup=True,

    # Route slow bulk work away from latency-sensitive jobs.
    task_routes={
        "app.worker.send_report": {"queue": "default"},
        "app.worker.reindex_account": {"queue": "bulk"},
    },
)


@celery_app.task(bind=True, max_retries=3, default_retry_delay=10)
def send_report(self, account_id: int) -> dict:
    """Pretend to build and deliver a report for an account."""
    try:
        time.sleep(5)  # stand-in for the slow upstream call
        return {"account_id": account_id, "status": "delivered"}
    except Exception as exc:  # noqa: BLE001 - narrow this in real code
        # Exponential-ish backoff via Celery's own retry machinery.
        raise self.retry(exc=exc)

Two settings do most of the reliability work. task_acks_late=True means a message is only acknowledged after the task returns, so a worker that dies mid-job leaves the message on the queue to be redelivered instead of silently losing it. worker_prefetch_multiplier=1 stops a worker from grabbing a batch of messages it then sits on while it grinds through one slow task — with long jobs, the default prefetch is how you end up with idle workers and a backed-up queue at the same time.

Late acknowledgement has one consequence you must design for: a task can run more than once. If the worker crashes after the side effect but before the ack, the redelivered message runs it again. Make the side effect idempotent, or key it so a repeat is a no-op — the same discipline covered in making POST safe to retry applies to background jobs verbatim.

Running the worker under systemd

In development, docker compose runs Redis and a worker together. In production, the worker runs as a supervised systemd service so it restarts on failure and starts on boot. Same image, same command, two ways of launching it.

# docker-compose.yml
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  worker:
    build: .
    command: celery -A app.worker.celery_app worker --loglevel=INFO --concurrency=4 --queues=default,bulk
    environment:
      REDIS_URL: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy

# /etc/systemd/system/async-jobs-worker.service
[Unit]
Description=Celery worker (async-jobs)
After=network-online.target redis-server.service
Requires=redis-server.service

[Service]
Type=simple
User=celery
Group=celery
WorkingDirectory=/opt/app
EnvironmentFile=/etc/async-jobs/celery.env
ExecStart=/opt/app/.venv/bin/celery -A app.worker.celery_app worker \
  --loglevel=INFO \
  --concurrency=4 \
  --queues=default,bulk \
  --hostname=async-jobs@%h
Restart=always
RestartSec=5
# Give in-flight tasks time to finish on stop before SIGKILL.
TimeoutStopSec=320

[Install]
WantedBy=multi-user.target

Running celery worker directly under Type=simple — rather than the older celery multi forking pattern — keeps systemd in charge of the process it supervises, so Restart=always and journal logging work the way you expect. Set TimeoutStopSec above your hard time limit so a graceful stop lets a running task finish rather than getting killed halfway. Enable it with systemctl enable --now async-jobs-worker and read its logs with journalctl -u async-jobs-worker -f.

The enqueue route: 202 and a status URL

The API’s job is to accept the work, return immediately, and give the caller a way to check on it. 202 Accepted is the honest status code here — the request was valid and queued, but the outcome does not exist yet. Return the task id and a Location header pointing at a status endpoint.

# app/main.py
from fastapi import FastAPI, Response, status
from pydantic import BaseModel

from app.worker import celery_app, send_report

app = FastAPI()


class ReportRequest(BaseModel):
    account_id: int


@app.post("/reports", status_code=status.HTTP_202_ACCEPTED)
def create_report(payload: ReportRequest, response: Response) -> dict:
    task = send_report.apply_async(args=[payload.account_id], queue="default")
    response.headers["Location"] = f"/reports/{task.id}"
    return {"task_id": task.id, "state": "queued"}


@app.get("/reports/{task_id}")
def report_status(task_id: str) -> dict:
    result = celery_app.AsyncResult(task_id)
    body = {"task_id": task_id, "state": result.state}
    if result.successful():
        body["result"] = result.result
    elif result.failed():
        body["error"] = str(result.result)
    return body

The status endpoint reads straight from the Redis result backend. result.state moves through PENDING, STARTED (because task_track_started is on), and then SUCCESS or FAILURE. One honest caveat: Celery cannot tell an unknown task id from a queued one — both report PENDING — so if you need to reject bogus ids, record the id in your own store at enqueue time and check there first. Put the same authentication in front of the enqueue route as any other write endpoint; an unauthenticated job submitter is a denial-of-service queue waiting to be filled, which is where disciplined API key scoping earns its keep.

What to watch once it is running

Two numbers tell you whether the setup is healthy: queue depth and task duration. If queue depth climbs steadily, you are enqueuing faster than the workers drain — add concurrency or workers, or move the slow bulk work onto its own queue and its own worker so it cannot starve latency-sensitive jobs. If task duration creeps toward your time limit, the limit is about to start killing real work, so either raise it deliberately or find out why the task got slower. Track failure and retry rates the same way you would track a deployment’s change-failure rate — a background job that fails silently is worse than one that fails loudly, because nobody is holding the request open to notice.

Start with this baseline, keep the side effects idempotent, and separate your queues before you need to — not after a bulk reindex has buried every user-facing job behind it.