Test Retry Budgets: Keep Retries Out of Assertions
A green rerun is not proof. Learn to set a test retry budget outside assertions, record seeds and outputs, and reproduce failures before changing code safely.
A Passing Rerun Is Not Proof: Put Test Retries on a Budget
A contract check fails: an order is still marked picking, but the test expects packed. You run the command again, and it passes. The temptation is familiar - add a retry to the assertion, make the suite "more resilient," and move on. But a green rerun doesn't tell you why the first run failed. It could be a worker that finished late, a warm cache, or leftover data from the previous attempt.
So the real question isn't "are retries bad?" It's where they belong, what they're allowed to prove, and what evidence they leave behind. In this worked example, a pairing session rejects putting the retry loop inside the test and instead records each attempt outside the assertion. The idea is intentionally modest: require a seed, cap the retry budget, capture a receipt, and reproduce the failure before you touch application code.
This is a proposed workflow, not a production incident report or a measured flake-rate study. It won't catch every timing bug. What it does do is make one common mistake harder: turning a lucky rerun into confidence that the contract is sound.
Why a green rerun can mislead you
Picture a service with GET /orders/:id. A test inserts one order, waits 200 milliseconds, then expects the response status to be packed. On the first run, the response says picking. On a second command (launched from the same shell without cleaning the database), you get packed and the command exits successfully.
That sequence shows the observed result changed. It doesn't automatically explain why. The second run might have benefited from a background worker finally finishing, or it might have read a row left behind by the first run, reused cached state, or landed in a different timing window. Calling it "eventual consistency" before you have supporting logs just gives you a story, not an explanation.
Ask about ownership before changing code
Before you edit a helper or increase a timeout, nail down three facts:
- Who owns the clock? Is the test waiting, is the application processing asynchronously, or is the database applying changes later?
- What is the seed? Can someone else recreate the fixture and the execution conditions?
- Can another process reproduce the result? Does a clean invocation behave the same way, or does the outcome depend on the first process's state?
These questions are more useful than starting with "How many retries should we add?" A retry count is a policy, not a diagnosis. It can't tell you which process was slow or whether the test saw stale state.
A rerun is a new observation, not a repair
In real projects, a passing rerun can still be useful evidence - it suggests the failure didn't happen under that later attempt. The catch is independence. If the second command shares a database, cache, filesystem, or long-lived worker with the first attempt, it's not a clean second data point.
That matters in CI just as much as on your laptop. If an assertion silently retries until it passes, the build can report success while hiding the original failure - and the number of attempts it took. Then the team loses the info needed to decide whether the cause was test isolation, asynchronous processing, or a real contract mismatch.
A green status is an output. It isn't an explanation.
Keep retry policy outside the assertion
A common patch wraps the assertion in a loop and returns the first response that looks right:
async function expectPacked(client, id) {
let last = null;
for (let attempt = 1; attempt <= 3; attempt += 1) {
last = await client.get(`/orders/${id}`);
if (last.body?.status === "packed") return last;
await new Promise((resolve) => setTimeout(resolve, 200 * attempt));
}
throw new Error("status never reached packed");
}
The code is small. The policy it creates is bigger than it looks. The test now decides how long to wait, how often to retry, and which result counts as success. Later, someone can raise the attempt limit and call the suite stable without adding evidence about the underlying behavior.
Tests should report a single attempt honestly
A single-shot assertion has one underrated benefit: it shows what happened during that invocation. If retries are hidden inside the assertion, the test result stops telling you whether the first attempt passed, whether the third attempt passed, or whether the test ran out of budget. That detail can matter when you're trying to compare CI failures with local reproduction.
This doesn't mean async systems should immediately fail when work is delayed. If the system is intentionally asynchronous, the contract should say so. Then the polling/retry policy needs an owner and a clear boundary - like a worker contract or a client-facing operation with a defined timeout. That's different from slipping an undocumented loop into a test assertion because one run was inconvenient.
Put the budget around the test process
The alternative is to run the test once and handle diagnostic reruns as a separate command. Then the retry budget is visible to the person running the workflow and can be recorded per attempt.
A practical rule set is:
- The test process runs the check once.
- Any rerun is a separate invocation with an explicit, bounded budget.
- Every attempt records the seed, command, exit code, and output hash.
- A second process or environment checks whether the same failure reproduces.
- No application patch is approved solely because a later attempt passed.
This isn't about banning retries. It's about stopping the test from silently deciding that a retry is proof.
Build a retry receipt
A receipt is a small record of what actually ran. It makes reruns auditable without pretending that a few attempts prove a statistical flake rate.
For each attempt, capture at least:
- The seed used to create or select the fixture.
- The exact command (or a stable identifier for it).
- The attempt number and the configured retry budget.
- The exit code.
- A hash of combined output, so you can compare results without storing everything forever.
- A short output tail only if it's safe to keep.
A possible receipt shape:
{
"decided": "retry_lives_outside_the_test",
"budget": 2,
"seed": "order-17-worker-not-run",
"attempts": [
{
"attempt": 1,
"exit": 1,
"body_sha256": "example-hash",
"tail": "status=picking"
},
{
"attempt": 2,
"exit": 0,
"body_sha256": "another-example-hash",
"tail": "status=packed"
}
],
"keep": "investigate_before_patch"
}
This is illustrative, not a stand-in for a recorded production result. The important parts are the seed, the attempt-level outcomes, and a decision that doesn't treat a pass as proof.
Fail closed when the seed is missing
A check that can't identify its input shouldn't be treated like meaningful reproduction. For a scratch script, make that explicit:
const seed = process.env.ORDER_SEED;
if (!seed || seed === "seed-not-set") {
console.error("seed missing; refusing to score a rerun");
process.exit(2);
}
Exit code 2 means the harness refused to run a valid comparison. It does not mean the product contract failed. Keeping those categories separate avoids turning "bad setup" into "failing behavior."
Treat the runner as tooling, not a production harness
A proposed Node.js receipt writer can run a check with child_process.spawn, pass a seed and attempt number via the environment, hash the captured output with SHA-256, and write a JSON file. That's enough to make a local experiment more legible - but it's not automatically safe or CI-ready.
In particular, anything that sets shell: true should only run trusted local commands. Don't point it at untrusted input. Validate that the retry budget is a positive integer; reject malformed values instead of letting them create weird loop behavior. Also think about whether captured output might include tokens, cookies, or customer data. If it could, keep the hash and drop the output tail.
And if you're using top-level await, use an .mjs extension or configure the package as an ES module. This is a scratch-repository workflow proposal, not code guaranteed to have been validated against a live service.
Interpret the evidence before patching
A receipt only helps if the team agrees on what it can and can't say. A retry budget of two is an operational limit for a diagnostic run. It's not a sample size for estimating how often the test flakes.
You can compare results from the current shell and then from a clean process environment. For example:
export ORDER_SEED="order-17-worker-not-run"
export RETRY_BUDGET="2"
export CHECK_CMD="node check-order.mjs"
node retry-receipt.mjs
env -i PATH="$PATH" HOME="$HOME" \
ORDER_SEED="$ORDER_SEED" RETRY_BUDGET=1 \
CHECK_CMD="node check-order.mjs" \
node retry-receipt.mjs
The second command helps remove some sources of leftover state, but env -i doesn't create a separate machine, database, or data volume. If the check still shares state with the first process, that state can still influence the result. For stronger isolation, set up an independent database or environment and draw that boundary clearly.
Use the outcome to choose the next step
Read the attempts as evidence, not as a popularity contest:
- The seed is missing and the check exits
2: the harness refused to score the run. Fix the setup before interpreting product behavior. - A dirty-shell run fails and then passes: leftover state or timing differences remain plausible. The pass doesn't authorize an application edit.
- A clean, single-shot process fails with
picking: preserve that failure and investigate which worker step should have moved the status. - Independent processes fail in the same way: the repeated mismatch is stronger evidence. Now inspect the worker path, fixture, and contract before proposing a patch.
- A receipt says
still_not_proof: respect the label. Exit code0doesn't upgrade itself into approval.
The "next step" depends on the system. If a worker is intentionally asynchronous, inspect its scheduling and completion contract. If the fixture is shared, isolate it. And if the API can legitimately respond before packing finishes, the contract may need to represent that state - rather than hiding it behind a longer test wait.
Keep receipts safe and the experiment narrow
Don't commit a receipt whose output tail includes secrets or customer payloads. Don't share a database between "independent" processes and then claim their results are isolated. And don't replace a stub with a live HTTP call before the stub workflow behaves as intended. Bringing a live service too early adds variables and makes the original disagreement harder to resolve.
This workflow also isn't a replacement for tracing, load testing, or deciding retry behavior at the system edge. It's a focused way to stop a test helper from laundering a lucky rerun into a merged change.
Conclusion: make the next run informative
When a contract check fails once and passes on a local rerun, the right reaction isn't automatically "add retries" or "ignore the failure." First, establish the seed, the process boundary, and what changed between attempts. Then run the assertion once, make diagnostic retries explicit, and keep a receipt that preserves the evidence without overstating what it proves.
This won't fix every async test problem. But it does make a common failure mode less likely: treating a green rerun as permission to patch (or permission to stop investigating). Try the receipt approach on a trusted scratch fixture, then adapt it to your own test and data boundaries before you use it in CI. For the original worked example, see the source article.
Disclosure: The source article was prepared as part of MonkeyCode product outreach and describes operator-supplied claims about free model access and a free server option. Those claims are not product endorsements or a verdict on the workflow. Check current project documentation; the retry-receipt rule does not depend on a particular assistant or hosting provider.