Long-Running API Requests Done Right: The 202 Accepted + Polling (or Callback) Pattern

A request that takes thirty seconds does not belong on the request path. It belongs to a background worker, and the client should be told to come back later.

The symptom is familiar. An export endpoint that ran fine against test data starts timing out in production. The load balancer cuts the connection at sixty seconds; the reverse proxy has its own thirty-second read timeout; the browser gives up; the mobile client retries and now the same expensive query runs twice. You add a timeout knob somewhere, it papers over the problem for a fortnight, and then a larger customer generates a larger export and you are back where you started. The fix is not a bigger timeout. It is to stop pretending the work is synchronous.

HTTP has had the right answer since the beginning: 202 Accepted. The request has been accepted for processing; the processing is not finished. You hand the caller a place to check on it, do the work off the request path, and let them fetch the result when it is ready. This post wires that up with FastAPI 0.115, Celery 5 and Redis 7 on Python 3.12.

The shape of the pattern

Three resources, not one. The POST that used to do the work now only accepts it: it validates the request, creates a job, dispatches it to a worker and returns 202 with a Location header pointing at a second resource — the status monitor. The client polls that status resource until the state is terminal. When it reports success, it carries a link to the third resource: the result itself.

  1. POST /exports → 202 Accepted, Location: /exports/{id}.
  2. GET /exports/{id} → the status resource: queued, running, succeeded or failed.
  3. GET /exports/{id}/result → the finished artefact, once the state is succeeded.

Keeping the status and the result as separate resources matters. The status document is small, cheap and polled often; the result may be large and is fetched once. Do not make clients download a megabyte of CSV on every poll just to learn the job is still running.

Model the state yourself, not Celery’s

The tempting shortcut is to expose Celery’s own task state through AsyncResult. Resist it. By default Celery reports PENDING for any task id it has not yet finished — including ids that never existed, because an unknown id and a queued id are indistinguishable to the result backend. It only emits a STARTED state if you switch on task_track_started. That is a leaky, ambiguous contract to hand a client.

Instead, keep a small status document in Redis that you control, written at each transition. Four states are enough: queued when the job is created, running when the worker picks it up, and succeeded or failed when it finishes. Give the document a TTL so results do not accumulate forever. If you have not already stood up the worker, the baseline Celery 5 and Redis setup covers the broker and backend wiring this builds on.

# tasks.py — the worker owns the status document
import json, time
import redis
from celery import Celery

celery_app = Celery(
    "exports",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)
celery_app.conf.result_expires = 3600  # Celery's own results, in seconds

_status = redis.Redis(host="localhost", port=6379, db=2)
STATUS_TTL = 3600  # how long a status/result stays fetchable


def write_status(job_id: str, **fields) -> None:
    key = f"export:{job_id}"
    current = _status.get(key)
    doc = json.loads(current) if current else {}
    doc.update(fields)
    _status.set(key, json.dumps(doc), ex=STATUS_TTL)


@celery_app.task(bind=True)
def build_export(self, job_id: str, params: dict) -> None:
    write_status(job_id, state="running", started_at=time.time())
    try:
        result = run_export(params)          # the slow work
        write_status(job_id, state="succeeded",
                     finished_at=time.time(), result=result)
    except Exception as exc:                  # record, then re-raise
        write_status(job_id, state="failed",
                     finished_at=time.time(), error=str(exc))
        raise

The three endpoints

The route layer stays thin. The POST writes the initial queued document, dispatches the task and returns 202 with the two headers a well-behaved client wants: Location for where to look, and Retry-After as a hint for how long to wait before the next poll.

# main.py
import json, uuid
from fastapi import FastAPI, HTTPException, Response
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from tasks import build_export, _status

app = FastAPI()


class ExportRequest(BaseModel):
    report: str
    from_date: str
    to_date: str


@app.post("/exports", status_code=202)
def create_export(req: ExportRequest, response: Response):
    job_id = uuid.uuid4().hex
    _status.set(f"export:{job_id}",
                json.dumps({"state": "queued"}), ex=3600)
    build_export.delay(job_id, req.model_dump())
    status_url = app.url_path_for("get_export", job_id=job_id)
    response.headers["Location"] = status_url
    response.headers["Retry-After"] = "2"
    return {"job_id": job_id, "state": "queued", "status_url": status_url}


@app.get("/exports/{job_id}", name="get_export")
def get_export(job_id: str):
    raw = _status.get(f"export:{job_id}")
    if raw is None:
        raise HTTPException(status_code=404, detail="unknown or expired job")
    doc = json.loads(raw)
    body = {"job_id": job_id, "state": doc["state"]}
    if doc["state"] == "succeeded":
        body["result_url"] = f"/exports/{job_id}/result"
    if doc["state"] == "failed":
        body["error"] = doc.get("error")
    pending = doc["state"] in ("queued", "running")
    headers = {"Retry-After": "2"} if pending else {}
    return JSONResponse(body, headers=headers)


@app.get("/exports/{job_id}/result")
def get_result(job_id: str):
    raw = _status.get(f"export:{job_id}")
    if raw is None:
        raise HTTPException(status_code=404, detail="unknown or expired job")
    doc = json.loads(raw)
    if doc["state"] != "succeeded":
        raise HTTPException(status_code=409,
                            detail=f"job is {doc['state']}, not ready")
    return doc["result"]

A few deliberate choices. The result endpoint answers 409 Conflict, not 404, while the job is still running — the resource exists, it is simply not ready. An expired or unknown id is a genuine 404. In production make the Location absolute rather than a path, so clients behind a proxy resolve it correctly. This is exactly the kind of contract worth pinning down in the schema first if you design the API spec-first with OpenAPI 3.1.

Polling, callbacks and TTLs

Polling is the default because it works everywhere and needs nothing from the client’s network. Honour Retry-After, and for longer jobs let clients back off rather than hammering the status resource every 200 milliseconds. If you control the caller and it is reachable, a webhook is the alternative: on the terminal transition, the worker POSTs the status document to a callback URL supplied at job creation. Treat that callback as fire-and-forget with retries, and keep polling available as the fallback — webhooks get lost, and the client still needs a way to recover the state.

The TTL is not an afterthought. Results held in Redis are memory you are paying for, so expire them on a schedule the client can live with — an hour is generous for an interactive export. Document that window, because a client that polls once a day and expects the result to still be there will be disappointed. And because the caller may retry the POST on a dropped connection, make job creation idempotent so a retry returns the existing job rather than starting a second expensive run; the same idempotency-key approach that makes POST safe to retry applies cleanly here.

None of this is exotic. It is the difference between an endpoint that falls over the first time real data arrives and one that scales with the worker pool behind it. If you are still assembling the surrounding application, the baseline FastAPI 0.115 project structure is where the routers above should live. Move the slow work off the request path, hand the caller a status resource, and the gateway timeout stops being your problem.

Free interactive tool

Website compliance checklist

What your site has to do, based on what it actually does

Answer as much or as little as you like — the list builds as you go. Nothing is stored against your name and no email is required.

Can you trust the architecture you have?

Architecture diagrams rarely show the reality of how systems actually operate. An independent review establishes what is really there.