CI Failures: Why Builds Pass Locally
Learn why code that passes locally fails in CI. Diagnose dependency drift, environment differences, missing files, and stale caches—and prevent repeat failures.
Why Your Code Passes Locally but Fails in CI
You push a commit that passes every check on your laptop. CI runs the same code and fails. Your first instinct is usually to blame the change. Sometimes it's right. But when the code is truly identical, the culprit is often something around it: a dependency that moved, an environment your laptop has quietly built up over time, a file Git didn't receive, or a cache handing your job an old artifact.
CI is often the first place your project gets tested from a clean checkout - fresh install, fresh setup, fewer leftovers. That's great, but it also exposes assumptions that local development can accidentally cover up. The goal isn't to prove CI is wrong or that your machine is right. The goal is to find the one difference that explains the result.
This guide covers four common causes of CI failures that aren't caused by the latest code change, and then lays out a repeatable order for diagnosing them. Along the way, you'll see how to reduce these failures without turning your build into a brittle, overengineered maze.
Start with the failure, not a theory
A failing CI job is evidence that a particular command behaved differently in a particular environment. By itself, it doesn't prove the code is defective, the platform is broken, or a dependency is to blame.
Start with the exact failing step. Read the job log and identify the command, the first meaningful error, and any file and line number in the annotation. A test summary that just says "one failed" isn't as helpful as the traceback right above it. The same goes for lint: a non-zero exit might be reporting a formatting or rules mismatch, not an application bug.
Then ask a few concrete questions:
- Did the job fail during dependency installation, compilation, linting, testing, or packaging?
- Does the failure point to a specific file, test, package, or tool version?
- Is CI running the same command and interpreter you use locally?
- Did anything outside the application code change (lockfile, workflow, base image, cache key)?
This keeps you from thrashing. If you immediately upgrade packages, clear every cache, and rewrite the failing test, you may erase the very clues that would have identified the cause. When possible, change one variable at a time and rerun the same failing command.
One useful distinction is between a code failure and a reproducibility failure. A clean environment can reveal a real defect that your laptop masked. It can also reveal that the project's dependencies or files weren't fully specified. Either way, the failure is telling you something about the system that runs the code - not just the code itself.
Four causes of CI failures outside your code
When the commit is unchanged but the result differs between a developer machine and CI, check these four areas first. They show up constantly because local setups make them easy to miss.
1. An unpinned or insufficiently constrained dependency
A dependency declaration like library>=1.4 lets the installer choose a newer "compatible" release. If a new version lands between your local setup and the CI run, the two environments can end up running different software - even if the repository hasn't changed.
That difference can be direct or transitive. You may depend on Package A, which depends on Package B. A new release of A might tighten requirements, and a release of B might change behavior your code relies on indirectly. Then the error appears in your code, while the real change came from deeper in the dependency graph.
A version range can limit that exposure. For example:
sqlmodel>=0.0.45,<0.0.48The lower bound sets a minimum version; the ceiling prevents the installer from quietly moving beyond the range you actually tested. What "right" constraints look like depends on your ecosystem and project, but in many setups a lockfile captures the resolved versions, while dependency manifests define what versions are acceptable.
There's a trade-off. Pinning everything to exact versions can make builds predictable, but it also forces you to manage updates intentionally. Loose ranges make upgrades easier to receive, but they also allow an unreviewed change into your build. A practical approach is to commit a lockfile where you can, use repeatable install commands, and keep a routine for updating dependencies.
When CI fails during installation or after what looks like a dependency change, compare the resolved package versions in CI with what you actually have locally. Don't assume the manifest version is what got installed.
2. The local and CI environments are different
"Same code" doesn't mean "same execution." Your laptop might run Python 3.12 while CI runs Python 3.11. Your local lint command might be using a globally installed tool, while CI installs the tool version declared by the project. And sometimes a leftover dependency from another project makes an incomplete dependency declaration appear to work.
CI starts from a defined image or runner and then runs the workflow setup steps. That clean start is valuable: it removes state accumulated over time. But it also means the workflow has to specify everything the project relies on - runtime, tools, environment variables, and setup commands.
Compare the details, not just labels like "latest" or "Python 3". Print the versions in both places:
python --version
python -m pip --version
python -m pip freeze
For other ecosystems, run the equivalent runtime and package-manager version checks. Also compare the exact command being executed. For instance, pytest from one shell might resolve to a different executable than python -m pytest. Using a project-specific virtual environment and explicit commands helps reduce that ambiguity.
Environment variables can cause the same kind of mismatch. Your local shell might export something CI doesn't set, or CI might provide a value your local tests never use. Treat configuration as part of the environment. If the failing step depends on it, check it.
3. A required file is not in version control
A test can pass locally because it reads a file that happens to be on your disk. If that file is ignored by Git, never added, or only exists due to an earlier build step, it won't be present in a fresh CI checkout.
This often involves fixtures, generated configuration, snapshots, local certificates, test data, or files excluded by a broad .gitignore rule. The failure can look unrelated: a parser complains about malformed input, a test can't find a fixture, or the app falls back to a default configuration.
Start from the failing test and list every file it reads. Then check what CI will actually receive:
git status --short
git ls-files path/to/needed-file
git ls-files tells you whether a path is tracked. If the file is intentionally generated, make sure the generation step runs before the test in CI. If it should be committed, add it and verify .gitignore isn't excluding it.
Don't "fix" this by committing secrets or machine-specific configuration. Credentials belong in the CI platform's secret store, not in the repo. For local-only settings, keep a safe example config in the repository and document how each environment gets its own values.
4. A cache is masking the actual state
Build and dependency caches save time by reusing artifacts from earlier runs. That's usually a good trade. The problem is that a broken or incomplete setup can look fine if CI restores artifacts that a clean install wouldn't produce.
A stale cache is one possibility, but it's not the only one. The cache key might be too broad, might ignore a lockfile or runtime version, or might restore generated output built for a different configuration. In some cases, the cache even hides a missing dependency because the package already exists in the cached environment.
Run a controlled comparison. Rerun the failing commit with the relevant cache disabled or invalidated, keeping the command and other inputs the same. If it fails without the cache but passes with it, the cache (or what it's restoring) needs attention. If the result is unchanged, the cache probably isn't the cause.
Avoid deleting every cache just because you're frustrated. That can make future runs slower without explaining anything. Instead, inspect the cache key and confirm it includes the inputs that affect the cached artifact (for example, your dependency lockfile and the relevant runtime version). A cache should speed up a correct build - not become an undocumented requirement.
Reproduce the CI environment locally
The quickest way to answer "is it code or environment?" is to reproduce the failing job as closely as you can. Exact parity won't always be possible - CI may run on hosted infrastructure you can't fully replicate. But matching the runtime, dependency resolution, and the command usually gets you close enough to find the difference.
Use a clean environment, not the virtual environment you've used for months. For a Python project, that can mean creating a new virtual environment, installing from your documented lockfile or requirements, and running the exact failing command. If it's container-based, run the same image and entrypoint CI uses. For other ecosystems, use the project's package-manager workflow and lockfile-based install approach.
A practical diagnosis order looks like this:
- Read the CI annotation and log. Identify the first failing command and the relevant file, line, or package.
- Compare runtime and tool versions. Check the interpreter, package manager, linter, test runner, and other tools involved in that step.
- Inspect dependency resolution. Compare what's actually installed, not just what the manifest says is allowed.
- Check repository contents. Confirm every required file is tracked or generated before the failing command.
- Run the same command from a clean setup. Remove accumulated local state, use the same versions CI uses, and don't rely on an existing cache.
- Change one variable and rerun. Disable a cache, adjust a version, or add a missing file - separately - so the result stays diagnostic.
If the failure reproduces in a clean local environment with the same inputs, move on to investigating the code, tests, or dependency compatibility. "Not your code" isn't a free pass to stop debugging - it usually means the immediate cause is outside the changed lines. Your app can still be relying on behavior that changed in a dependency, or on an environment assumption that never got made explicit.
If the failure doesn't reproduce, then compare what's still different: the CI image, environment variables, operating system behavior, filesystem differences, network access, and job ordering. A mismatch between Linux CI and macOS development, for example, can expose case-sensitive file paths that your local filesystem tolerates. Keep the scope tight and start with the most likely difference before rewriting the workflow.
Prevent repeat failures without overengineering
You don't need every machine to be identical down to the last detail. It's expensive, and it usually isn't necessary. What you do need is to make the inputs that matter visible, repeatable, and easy to compare.
Define the build inputs that affect the result
For most applications, the inputs include the language runtime, dependency versions, build and test commands, required configuration, and any generated files. Put those decisions in repository-managed files or in a clearly documented CI workflow. Don't rely on setup that only exists on one developer's machine.
Use lockfiles and reproducible install commands when your ecosystem supports them. Pin or constrain tools when version changes can affect linting, compilation, or tests. This doesn't mean freezing the project forever. It means dependency and tool updates happen as a deliberate, reviewable change, not as a surprise during an unrelated CI run.
Make the CI command runnable by developers
A workflow is easier to debug when its key commands can run locally too. Prefer a documented project command or script over a long chain of shell-specific steps hidden in CI config. If CI runs lint, tests, and a build, make it clear how to run those same checks from a clean checkout.
You don't have to force everyone onto the same operating system. The point is to reduce accidental differences in command selection and setup. If CI includes a legitimate platform-specific step, keep that distinction explicit.
Treat caches as performance optimizations
A correct build shouldn't rely on a cache being present. Make cache keys reflect the inputs that determine the cached content, and periodically verify a clean run still succeeds. When a failure appears cache-related, test it with a targeted cache bypass instead of repeatedly clearing unrelated caches.
The same idea applies to generated artifacts. If the build needs a generated file, your workflow should generate it as part of the build or restore it from a deliberately managed artifact. A leftover file from an earlier run shouldn't become a hidden prerequisite.
Choose predictability at the right level
Not every project needs a containerized dev environment, fully pinned toolchains, and a heavy dependency update pipeline. Those controls cost time to build and keep running. In smaller teams, a lockfile plus a documented runtime version might be enough. In a bigger project with frequent releases and many contributors, stronger environment controls can pay off quickly.
Pick the amount of "machinery" that matches the cost of failure. If an unexpected dependency update can break releases, invest in tighter constraints and automated update checks. If a tool is low-risk and easy to update, a wider version range might be reasonable. The goal is to make that trade-off on purpose, not inherit it by accident.
Conclusion
A red CI job doesn't automatically mean the latest code is broken. It means the build ran under conditions that produced a different result. Dependency drift, environment differences, untracked files, and misleading caches are practical places to start when the commit is unchanged.
Read the failure first. Compare the actual inputs, then reproduce the failing command in a clean environment. Fix the underlying assumption: constrain or lock dependencies, define the runtime, track or generate required files, and revisit the cache strategy. Keep the solution proportional to your project. A predictable build helps everyone; a complicated build system nobody can maintain does not.
When CI fails again, resist guessing. Follow the evidence until you can explain which input changed and why. That explanation - not just a green rerun - is what makes the next failure faster to solve.