Failing-Test Signature Refactor: 5 Steps ⚡ (Git Worktree)
Learn failing-test signature safe refactoring in 5 steps: use test failure analysis, characterization testing, and Git worktree to spot coupling and land changes fast.
Find Refactor Coupling Before You Commit
A small diff isn't automatically a safe change. In a mature codebase, a two-line edit can break a contract shared across several modules. Meanwhile, a bigger extraction that's contained well might have little impact elsewhere. Diff size tells you how much text changed - not which changes depend on the same behavior or state.
A better question is: Which tests fail when you apply each proposed edit by itself? The set of failing tests is the failure signature. Compare those signatures across small, single-intent patches, and you can spot where edits are likely touching the same behavior. That's useful evidence for what should go into one commit versus what can land independently.
This isn't a proof of safety, though. A green test suite can miss real behavior, and failures can be noisy or misleading. Still, for a large refactor, measuring coupling in a disposable worktree is often cheaper than trying to explain a deliberately broken intermediate commit during review. This guide covers a practical workflow, how to interpret what you see, and the limits you should keep in mind before you trust the matrix.
Why Test Signatures Help With Refactors
Characterization tests capture existing behavior. They help when you want to change an implementation without accidentally changing what users - or other parts of the system - rely on. But simply passing those tests doesn't tell you which planned edits are coupled.
Take a refactor that renames a field used by both a producer and a consumer. The rename is one intent, repeated in two places: update the contract everywhere. If you split those edits into separate commits, the first commit can leave the app in a broken state. Reviewers then have to reason about a failure that was never intended to represent a valid intermediate version.
The real problem isn't that every commit must be independently releasable. Some teams accept short-lived red commits in a branch. The issue is that an unexplained red commit makes review and troubleshooting harder - especially when the failure would have been easy to catch before pushing.
Treat failures as evidence of shared behavior
For each candidate patch, run the same test suite against a clean baseline and record which tests fail. That list is the patch's signature. If two patches fail the same tests, they may be affecting shared state, a shared contract, or a common dependency.
"May" matters. A shared failing test is a reason to investigate, not proof that the changes must be bundled. Likewise, different failures don't prove independence. The tests only show coupling that the suite can actually observe.
That's also why the workflow starts with a trustworthy baseline and small patches. Without those, you're mostly comparing noise: inconsistent test runs, broad changes, or edits that aren't really separable into distinct intents.
Group by intent, not by file
A patch should represent one sentence you could put in a commit message: extract a helper, rename a field, move a validation call. Don't group work just because edits happen to land in the same file.
One file can contain unrelated changes. One intent can span multiple files. Keeping patches small makes the failure matrix (and the eventual review) easier to reason about. If a reviewer can't understand the patch in a minute or two, it's probably too broad for this method.
A Practical Workflow for Measuring Coupling
This workflow has five stages: pin the baseline, make single-intent patches, apply and test each patch in isolation, compare signatures, and land one group at a time. The repeated work is intentional - you want to find coupling before it becomes part of a long commit history.
1. Pin a clean, green baseline
Start from a clean working tree and a test suite that passes. Record the exact commit and the exact test command. Don't quietly change test selection or configuration between runs, or the signatures won't be comparable.
mkdir -p .matrix
git status --short
git rev-parse HEAD > .matrix/baseline.sha
python -m pytest -q --tb=no
Save the command in a small script or text file, then reuse it for every patch. If you need a JUnit report to extract test IDs, generate it consistently and verify the report path doesn't change what tests run.
If there aren't meaningful tests for the behavior you're changing, add characterization tests first. A few focused tests around public entry points are usually more useful than a big suite that mostly asserts implementation details. For more on pytest's report options, see the pytest documentation on test reports.
2. Split the work into single-intent patches
Create patches against the same baseline. Each patch should apply on its own and express one change in intent, even if it touches multiple files.
patches/
01-extract-normalize.patch
02-rename-customer-id.patch
03-move-validation.patch
Make sure your working tree is clean before generating or applying patches. If a patch can't apply to the baseline without another patch, record that dependency. It's already coupled textually, even before you run tests.
A patch that applies by itself can still be behaviorally coupled to another patch. That's why the next stage matters.
3. Apply each patch in a disposable worktree
Don't reset your main working tree over and over to test candidates. A Git worktree lets you run each test in a separate checkout you can delete afterward. That keeps experiments isolated from your in-progress changes.
Below is a simplified example. Adapt the repository path, patch format, and test command to your project. It assumes pytest can produce a JUnit XML report and that test case identifiers are stable.
import pathlib
import subprocess
import sys
import xml.etree.ElementTree as ET
repo = pathlib.Path('.').resolve()
scratch = repo / '.matrix'
patches = sorted((repo / 'patches').glob('*.patch'))
scratch.mkdir(exist_ok=True)
def run(args, cwd):
return subprocess.run(args, cwd=cwd, capture_output=True, text=True)
def failing_tests(report):
failed = set()
for case in ET.parse(report).iter('testcase'):
if case.find('failure') is not None or case.find('error') is not None:
failed.add(f"{case.get('classname')}::{case.get('name')}")
return frozenset(failed)
def signature(patch):
worktree = scratch / f'wt-{patch.stem}'
added = run(['git', 'worktree', 'add', '--detach', str(worktree), 'HEAD'], repo)
if added.returncode != 0:
return 'WORKTREE_ERROR'
try:
applied = run(['git', 'apply', str(patch)], worktree)
if applied.returncode != 0:
return 'DOES_NOT_APPLY'
report = worktree / 'report.xml'
result = run([
sys.executable, '-m', 'pytest', '-q', '--tb=no',
f'--junitxml={report}'
], worktree)
if not report.exists():
return 'NO_REPORT'
return failing_tests(report)
finally:
run(['git', 'worktree', 'remove', '--force', str(worktree)], repo)
for patch in patches:
print(patch.name, signature(patch))
In a production version, record the test command, baseline commit, exit status, and logs with each result. The snippet shows the core idea; it isn't a full test orchestration framework. Confirm the test runner's exit code is successful or at least understandable, and treat an incomplete or missing report as an error - not as "no failures."
4. Compare signatures, then investigate
You can use Jaccard similarity to measure overlap between two failure sets: the number of shared failing tests divided by the total number of distinct failing tests across both sets. A higher score means more overlap, but it's a heuristic, not a magic threshold for bundling changes.
Don't treat two empty signatures as evidence of coupling. Depending on how you compute similarity, empty vs empty can end up as "perfectly similar." That would group every patch that produces no observed failures. For this workflow, either exclude empty signatures from clustering or define their similarity as 0. An empty signature means "no failure observed," not "these edits share behavior."
Also be careful with transitive grouping. If patch A overlaps patch B and patch B overlaps patch C, that doesn't automatically mean A and C should be in the same commit. Look at the actual failing test IDs and the intent behind each patch before merging groups.
5. Land one group, then refresh the baseline
Apply one reviewed group, run the full relevant suite, and only commit when the result is acceptable. Then update the baseline and re-measure the remaining patches. Coupling can shift as code moves, so an old matrix isn't a permanent map of the repository.
git apply patches/01-extract-normalize.patch
python -m pytest -q --tb=no
git commit -am 'refactor: extract normalize'
git rev-parse HEAD > .matrix/baseline.sha
The details of applying a group depend on how you manage patch files and branches. The principle is consistent: keep the tested baseline clear, land a coherent unit, verify it, then rerun the remaining analysis on the new state.
How to Read the Failure-Signature Matrix
Treat the matrix as evidence, not an automated verdict. Start with the failing test names, then trace those tests back to the code and the intent of each patch.
| Signature shape | What it suggests | Practical response |
|---|---|---|
| Empty | No coupling visible to the current suite | Likely independent, but keep normal review and testing |
| Identical, non-empty | Both patches expose the same failing behavior | Investigate shared state or contract; consider one commit |
| Partial overlap | Some behavior may be shared, while other failures differ | Inspect the overlapping tests and decide whether order matters |
| Disjoint, non-empty | Each patch has a different observed impact | Investigate separately; don't assume either change is safe |
| Does not apply | A textual dependency exists | Stack or regenerate the patches against the right base |
| Missing or invalid report | The run didn't produce usable evidence | Fix the test or reporting setup and rerun |
For example, suppose a field rename and a validation move each fail the same seven tests when applied alone. That overlap is a reason to inspect the shared contract and probably keep the changes together. On the other hand, if a helper extraction produces no failures, it might be a reasonable early commit - but only if the tests genuinely exercise the affected behavior.
A useful rule of thumb: trust specific overlap more than intuition, but still check what the overlap actually means. The matrix helps answer "Where should I look?" It doesn't answer "What is the correct design?"
Use stable test identities
Failure signatures depend on test IDs. If names change between runs, parametrized tests generate unstable identifiers, or reports omit class names, differences might just be reporting artifacts. Check a few report entries manually before you automate comparisons across a large patch set.
Also keep test configuration stable. Different environments, random seeds, external services, or timestamps can produce failures that have nothing to do with your patch. A noisy signature isn't a measurement of coupling - it's a measurement of noise mixed with coupling.
Limits, Costs, and When to Skip the Method
The cost is fairly simple: one test run per patch, plus the time to create and review those patches. If a suite takes two minutes and you have thirty candidates, the runs alone can eat up about an hour. That's often fine for a risky refactor. It's overkill for a tiny fix with an obvious blast radius.
You can reduce cost by running a reliable package-level or test-selection target first, then running the full suite when you're ready to land a group. This only helps if the smaller target actually covers the code you're changing. Faster feedback isn't useful if it gives you false confidence.
The test suite defines what the matrix can see
Tests that pass without asserting relevant behavior will give you empty signatures for changes that are still coupled. Brittle tests can also fail on harmless edits and produce broad signatures. Golden files that include timestamps, unstable ordering, or environment-specific values often cause misleading failures.
This method can't protect behavior you never pinned with tests. It also can't detect semantic drift that your tests accept. Treat empty signatures as a lower bound on observed coupling, not as a guarantee of independence. If tests are unreliable, fix them before leaning on the matrix.
When this workflow is a poor fit
Skip the full process if you'll run out of time, if you can't add characterization tests, or if the change is a one-line fix with an obvious, contained effect. This method is most useful when the module is large, the suite is credible, and reviewers keep running into intermediate commits that fail for unclear reasons.
Automation can help generate candidate patches or run the worktrees, but it doesn't replace judgment. You still need to inspect the edits, validate the scope, and interpret the failures. In the source workflow, candidate patches were drafted with a model and most were discarded. That's a sensible use of assistance: drafting is cheap, but deciding what the evidence means is the real engineering work.
The source material also mentioned that the article was prepared as part of MonkeyCode product outreach, including free model access and a free server option. Those tools might help with patch drafting or repeated runs if they fit your needs. They're optional. You can run the same workflow locally, and hosted services don't make a weak baseline any more trustworthy.
Conclusion: Make Each Commit a Known Unit
The smallest safe change isn't always the smallest diff. It's the smallest set of edits whose failure modes you understand well enough to review and land with confidence.
Pin a green baseline, split the refactor by intent, test each patch in an isolated worktree, and compare the resulting failure signatures. Treat the matrix as a guide for investigation - not as a safety certificate. Then land one coherent group, rerun the suite, and refresh the baseline before continuing.
If reviewers are spending time debating red intermediate commits, try this on the next large refactor. Start with a few representative patches and a trustworthy test target, then expand the workflow only if the evidence saves more time than the test runs cost.