Why Did My Pipeline Pass Locally but Fail in CI? The Usual Suspects

The build is green on your laptop and red in CI, and the diff is empty. This is not a mystery, it is a short list of usual suspects, and working through them in order turns a ghost failure into a five-minute diagnosis.

Nobody enjoys the “works on my machine” failure, because it feels like the machine is lying to you. It isn’t. Your machine and the runner are simply two different computers with two different states, and the pipeline is faithfully reporting a real discrepancy. The job is to find which of a small number of well-known discrepancies you have hit. Almost every case falls into one of five buckets: files the runner never received, environment the runner never had, state that leaked between test runs, a filesystem that treats letter case differently, and a clock or locale set somewhere else. Go through them in that order.

Suspect one: files the runner never saw

The single most common cause is the most boring. The runner checks out exactly what is committed and pushed, nothing more. That untracked fixture, that config file you created weeks ago and never added, that generated migration sitting in your working tree, that dependency you installed globally with a one-off command, none of it exists on the runner. Your local pass depends on it; CI cannot.

Prove it before you theorise. A clean clone into a temporary directory reproduces exactly what the runner starts from.

# See anything Git is not tracking or is deliberately ignoring
git status --porcelain --ignored

# Reproduce the runner's starting point: a pristine checkout of what you pushed
tmp=$(mktemp -d)
git clone --depth 1 --branch "$(git rev-parse --abbrev-ref HEAD)" \
  "file://$(git rev-parse --show-toplevel)" "$tmp/repo"
cd "$tmp/repo" && ls -la

If your tests fail in that clean clone, you have found it locally, without waiting for a CI round trip. A committed .gitignore that is too aggressive causes the same symptom in reverse: a file you need is being ignored and never gets added. This is exactly the class of drift a committed pre-commit baseline that actually runs is meant to catch before the push, not after the red build.

Suspect two: environment the runner never had

Your shell has years of accumulated state. Exported variables in .bashrc or .zshrc, a .env file your test runner loads automatically, credentials in your keychain, a Python virtualenv with a different interpreter version than the one CI pins. The runner starts cold. If a test reads DATABASE_URL or an API token and passes locally, ask where that value comes from and whether the runner has it. Secrets in particular should be injected explicitly through the CI platform’s secret store, never assumed, and never the same long-lived value on both sides. Treating them as short-lived and scoped, the way you would when you make API keys that are not a liability, removes a whole category of environment surprise.

The version gap deserves its own line. “Python 3.12 on my Mac, 3.11 in the workflow file” produces failures that look like your code and are really an interpreter difference. Pin the versions in the workflow, and match them locally.

Suspect three: state that leaks between tests

If the failure is a test rather than the build, order is the prime suspect. Locally you often run one file, or run the suite in the same order every time, so a test that quietly depends on another test having run first, seeding a database row, writing a temp file, populating a module-level cache, keeps passing. CI runs the full suite, sometimes sharded across workers, sometimes shuffled. The hidden dependency surfaces.

Do not guess at which test is dirty. Randomise the order and let the seed tell you.

# pytest-randomly ships a seed with every run; a failure is now reproducible
pytest -p randomly

# Reproduce a specific bad ordering the runner reported
pytest -p randomly -p randomly --randomly-seed=1234567890

# Confirm a single test stands on its own, with no warm cache or leaked rows
pytest tests/test_billing.py::test_invoice_totals --cache-clear

Once you can reproduce the order, the fix is almost always a missing fixture or teardown, not a race. A test that needs a row should create it; a test that mutates global state should restore it. Order-independence is the property you are really after.

Suspect four: the filesystem that reads case

This one is vicious because it is invisible on the machine where you write the code. The default macOS filesystem and Windows are case-insensitive: import UserService resolves a file named userservice.py without complaint. The Linux runner’s filesystem is case-sensitive, and the same import fails with a module-not-found error that makes no sense next to a green local run. Git makes it worse: with core.ignorecase set to true, a rename that only changes case may not even be recorded, so the wrong-cased name is what the runner checks out.

# What does Git think about case on this machine?
git config core.ignorecase

# Find names that collide only by case, exactly what Linux will trip over
git ls-files | tr 'A-Z' 'a-z' | sort | uniq -d

# Correctly record a case-only rename that a case-insensitive FS hides
git mv --force UserService.py userservice.py

Make the check part of the pipeline itself. A pipeline that flags case collisions costs nothing and closes off an entire recurring failure mode for every developer on a Mac.

Suspect five: clocks, locales and the machine itself

GitHub-hosted runners operate in UTC. Your laptop does not. A test that formats a timestamp, asserts on a date boundary, or parses “today” passes in your timezone and fails at a certain hour on the runner. Locale is the same story: number and date formatting, and string sorting, all shift with LANG and LC_ALL. Pin them so both machines agree, and set the expectation explicitly rather than inheriting the host’s.

When the first four suspects come up empty, stop reasoning by proxy and run the runner’s own image. Docker gives you the actual environment, and act runs the workflow file itself against a container that mirrors the hosted runner.

# Pin the environment the way the runner does, then run your suite inside it
docker run --rm -it \
  -e TZ=UTC -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 \
  -v "$PWD:/src" -w /src \
  python:3.11-slim bash -lc "pip install -e . && pytest"

# Or run the GitHub Actions workflow itself, locally, against a runner image
act push -P ubuntu-latest=catthehacker/ubuntu:act-latest

If it fails there, you have finally moved the failure onto a machine you can poke at with a debugger and a shell, which is the whole point.

The checklist, and why it matters beyond the one build

In order: clean-clone to rule out uncommitted files; diff the environment and pin versions and secrets; randomise test order with a recorded seed; scan for case collisions; and pin timezone and locale, then reproduce inside the runner image. Nine failures in ten are named in that list, and the tenth is usually a genuinely flaky external dependency, which is its own conversation.

There is a governance point underneath the engineering one. A pipeline that fails for reasons nobody can quickly explain trains a team to ignore red, and a team that ignores red loses the signal the pipeline exists to give. The health of this feedback loop is measurable, and change-failure rate is one of the numbers worth instrumenting alongside the rest of the DORA four. A ghost failure you can diagnose in five minutes is a rounding error. A ghost failure nobody can explain is how a delivery organisation quietly stops trusting its own build.