Why Did My Celery Task Execute Twice? Visibility Timeout, acks_late and Idempotent Tasks

A customer received two identical confirmation emails from the same order. The application logs show one call to the task. The worker logs show it ran twice, eleven minutes apart. Nothing in your code sent it twice — the broker did.

This is the single most common surprise people hit with Celery in production, and it is not a bug. It is the delivery guarantee working exactly as designed. Celery, on every mainstream broker, gives you at-least-once delivery. That means the framework will happily run your task more than once under a defined set of conditions, and it is your job — not the broker’s — to make that safe. If you assumed exactly-once, you built on a guarantee that was never offered.

Let me walk through the two mechanisms that actually cause the duplicate, how to reproduce them, and the only fix that holds: an idempotent task.

Cause one: the visibility timeout expired

When you use Redis as the broker, there is no native acknowledgement channel the way there is with a real message queue. Celery emulates one using a visibility timeout. When a worker picks up a message, the broker does not delete it — it hides it for a fixed window. If the worker acknowledges completion within that window, the message is removed. If the window elapses first, Redis assumes the worker died and makes the message visible again, so another worker collects it. Your task runs a second time.

The default visibility timeout on the Redis transport is 3600 seconds — one hour. That sounds generous until you meet the real failure mode. With late acknowledgement enabled (below), the message is only acknowledged after the task returns. A task that legitimately takes longer than the visibility timeout — a large report, a slow third-party API, a batch that grew — will be redelivered while the first copy is still running. Now you have two workers executing the same job concurrently. This is the classic redelivery loop, and long-running tasks make it worse, because each copy also exceeds the timeout and triggers yet another redelivery.

You tune it in broker_transport_options, and it must exceed the longest possible runtime of any task on that broker, including retries and back-off:

# celeryconfig.py  (Celery 5, Redis 7)
# The visibility timeout must be longer than your slowest task,
# otherwise Redis redelivers a message that is still being processed.
broker_transport_options = {
    "visibility_timeout": 43200,  # 12 hours, in seconds
}

This raises the ceiling; it does not remove the risk. A worker that is killed mid-task will still cause a redelivery once the (now longer) window passes. Tuning the timeout buys you correctness for well-behaved tasks. It does not make the task safe to run twice.

Cause two: acks_late redelivered after a crash

By default Celery acknowledges a message early — the moment the worker reserves it, before the task body runs. That protects you from duplicates, but it loses work: if the worker dies mid-task, the message is already gone and the job never completes.

To avoid losing work, people set task_acks_late = True (or acks_late=True on the task). Now the acknowledgement happens after the task returns. If the worker is killed — an out-of-memory kill, a deploy that sends SIGKILL, a spot instance reclaimed — the message was never acknowledged, so the broker redelivers it and another worker runs it. That is exactly the behaviour you asked for: no lost tasks. The unavoidable cost is that a task which had already done its work, but crashed before acknowledging, runs a second time.

So the two knobs pull against each other. Early acknowledgement risks losing work; late acknowledgement risks running work twice. There is no setting that gives you exactly-once — that guarantee does not exist across a network with failures. The honest position is: you will get at-least-once, so design for it.

Reproducing it deliberately

You can force the redelivery in under a minute. Set a short visibility timeout, enable late acknowledgement, and write a task that sleeps past the window:

# tasks.py  — reproduces a redelivery on Redis
import time
from celery import Celery

app = Celery("demo", broker="redis://localhost:6379/0")
app.conf.broker_transport_options = {"visibility_timeout": 10}
app.conf.task_acks_late = True

@app.task
def slow_job(order_id):
    print(f"running slow_job for {order_id}")
    time.sleep(25)  # longer than the 10s visibility timeout
    print(f"finished slow_job for {order_id}")

Start a worker, call slow_job.delay(42) once, and watch the log: a second "running" line appears roughly ten seconds in, while the first is still sleeping. One call, two executions. If slow_job sent an email, your customer got two.

The real fix: make the task idempotent

Since you cannot prevent redelivery, make a second execution a no-op. The durable way to do this is a dedupe guard in the database: claim a unique key for the unit of work, and let the database reject the second claim. PostgreSQL’s INSERT ... ON CONFLICT DO NOTHING does this atomically, so two concurrent workers cannot both win. This is the same discipline as an HTTP idempotency key that makes a POST safe to retry — applied to the task layer.

-- one-off migration (PostgreSQL 17)
CREATE TABLE task_dedupe (
    idempotency_key text PRIMARY KEY,
    created_at      timestamptz NOT NULL DEFAULT now()
);

# idempotent_tasks.py  (Celery 5, PostgreSQL 17, Python 3.12)
import psycopg  # psycopg 3
from celery import Celery

app = Celery("demo", broker="redis://localhost:6379/0")
app.conf.task_acks_late = True

DSN = "postgresql://app:secret@localhost:5432/app"

def claim(key: str) -> bool:
    """Return True only for the first caller to claim this key."""
    with psycopg.connect(DSN, autocommit=True) as conn:
        cur = conn.execute(
            "INSERT INTO task_dedupe (idempotency_key) VALUES (%s) "
            "ON CONFLICT DO NOTHING",
            (key,),
        )
        return cur.rowcount == 1  # 1 = we inserted, 0 = already claimed

@app.task(bind=True, acks_late=True)
def send_confirmation(self, order_id: int):
    # Derive a stable key from the work, not from a random per-call value.
    key = f"order-confirmation:{order_id}"
    if not claim(key):
        # A previous (or concurrent) delivery already handled this order.
        return "duplicate suppressed"
    _send_email(order_id)
    return "sent"

The key must be derived from the work itself — the order ID, an invoice number, a content hash — not from a value generated inside the task, or every delivery mints a fresh key and the guard never fires. Where the side effect is itself a database write, prefer collapsing the whole operation into a single transaction that is naturally idempotent (an upsert keyed on the business identifier), so the guard and the effect commit together.

Two operational notes. First, the guard claims before the side effect, which means a crash between the claim and the send can suppress a legitimate retry — for genuinely critical effects, record the outcome and reconcile, rather than trusting the claim alone. Second, the dedupe table grows forever unless you prune it; a scheduled job deleting rows older than your maximum retry horizon keeps it bounded.

What to take from this

Duplicate execution is not an exotic edge case you can configure away. It is the standing behaviour of an at-least-once system, and it will surface the first time a worker is redeployed under load or a task runs long. Tune visibility_timeout above your slowest task, decide deliberately between early and late acknowledgement knowing what each one costs, and make every task with a real-world side effect idempotent. This belongs in the tests you enforce in CI, alongside the rest of your automated baseline — a redelivery test that asserts the second run is a no-op is cheap to write and catches the regression that would otherwise reach a customer.

The duplicate email is a nuisance. The duplicate payment, the double refund, the twice-charged invoice is an incident — and it counts against your change-failure rate whether or not anyone traces it back to a broker redelivery. Design the task so running it twice is boring, and you never have to.