Pytest Output Filtering for AI Coding Agents

Why raw test logs become an agent problem

A test runner is built to help a developer inspect what happened. An AI coding agent has a different job: it needs enough evidence to identify the failure, understand the relevant rule, make a change, and verify it worked. Those goals overlap, but they aren't the same.

A verbose pytest run might print 76 passing cases one by one, then show four failures. It can also include repeated setup messages, captured debug output, color escape sequences, and stack frames from pytest internals. Humans can scan to the failures or jump to them. An agent typically gets the whole thing inside a prompt, where every character competes for context and attention.

The practical cost is cumulative. If a tool response stays in conversation history, later turns keep dragging it along. Another test run adds another blob rather than replacing the previous one. Your agent's next prompt can grow faster than the actual work it's doing. A small bug fix may end up consuming tokens on old output - including "everything that passed" lines that didn't change.

There's also a quality problem. More text isn't always more useful text. A wall of logs can bury the few lines that separate a boundary error from an ordering bug. If the agent's view is incomplete or noisy, it may patch the most obvious failure, rerun tests, and only then realize there were related failures underneath. That means extra calls and extra verification cycles.

The fix isn't "hide all output." Failures need enough detail to diagnose. Instead, separate the test runner's full output from the concise result you send to the model. Keep raw logs available to your tool (or stored as an artifact) when needed, and send the agent a structured summary by default.

The core idea: humans often want a complete transcript to review. The model usually needs a compact report of what failed, where, and what the failure says. Build the tool response for the next decision - not for terminal nostalgia.

What the benchmark measured

The benchmark compared two harness modes on an 80-test payment-settlement suite: 76 tests passed and four failed across validation, settlement, and statement modules. Both modes ran in the same isolated virtual machine with the same model and task. The main difference was raw terminal output versus a deterministic preprocessing step that returned a JSON failure summary.

In the raw mode, verbose pytest output was appended to the conversation. The first test response was 60,749 characters. Over the run, cumulative CLI output reached 117,391 characters. In the filtered mode, ANSI sequences, environment banners, passing-test lines, and framework noise were removed, and the tool returned a 1,506-character JSON report on the first run. Cumulative output reported for that mode was 1,604 characters.

Here's what the comparison showed:

In the raw-output run, the second turn's input reached 31,170 tokens. After another test run, the third turn reached 58,809 input tokens. The agent made a partial fix before addressing all three modules. With the filtered report, the four failures and their expected behavior were visible together, and the agent finished the fix in the second call.

So the mechanism isn't magic - it's less repeated context and a clearer view of the whole failure set. The filter didn't make the model "smarter." It just made the evidence easier to use.

There are limits to what this proves. It's one benchmark, one repository, one model configuration, and a hidden suite of 20 cases. The result supports structured output in your own harness, but it doesn't claim every agent, project, or test runner will hit the same 94.1% savings. It also doesn't independently validate the performance of any specific commercial filter product. The measured change came from the harness behavior described in the benchmark.

What a useful test filter should preserve

A filter is only useful if it removes repetition without removing diagnosis. A count of failures alone is too aggressive. Reposting the original traceback unchanged defeats the point. What you want is the minimum evidence the agent needs to locate the issue, infer what rule was violated, and pick a correction.

At minimum, your report should include:

• Run status: total, passed, failed, and skipped counts, plus the process exit code.
• Failure identity: test name, test file, and the relevant source location when available.
• Failure evidence: the assertion message and expected-versus-actual values, not just a generic "failed" label.
• Useful context: a short excerpt of the application-level traceback or captured output when it affects diagnosis.
• A path to more detail: store the full log somewhere your tool can retrieve, or provide a follow-up command when the summary isn't enough.

For example, a compact report might look like this:

{
"summary": {
"total": 80,
"passed": 76,
"failed": 4,
"filtered_noise_chars_removed": 59243
},
"failures": [
{
"test": "test_validator_contract",
"file": "validator.py",
"line": 14,
"assertion": "Expected JPY to be allowed; empty lists and booleans must be rejected."
}
]
}

In the benchmark, the failure descriptions weren't just "an assertion happened." They surfaced behavioral rules like preserving first-seen settlement order, applying a reserve only above a threshold, and retaining a minus sign for zero values in a statement. That's the kind of semantic detail that a test name alone can't always carry.

Don't rely on brittle text matching if your test framework can produce machine-readable results. A JSON report (or a JUnit-style XML report) is a better boundary than trying to infer failures from colored terminal output. You may still need a small adapter to attach concise assertion details and source locations.

A good design is layered: send a concise summary in the normal agent loop, but keep the full report around for when the agent asks. That preserves diagnostic depth without charging your model for every passing-test line on every routine turn.

How to add structured output to your test loop

Treat test execution like a tool call with an output contract, not like a shell command whose stdout gets pasted wholesale into chat. Your wrapper should run the tests, collect status, build a compact report, and return that report to the agent. Keep the full output separate so you can inspect it when the summary isn't clear.

A practical implementation sequence:

1. Capture output at the harness boundary. Run the test process from your tool wrapper and retain stdout, stderr, and the exit code. Don't append them directly to the model response by default.
2. Prefer a machine-readable reporter. Configure pytest to emit structured results, or use a reporter that produces JSON or XML. A quieter invocation helps, but it isn't a substitute for structured failure data.
3. Normalize the report. Remove terminal control codes and repetitive environment banners. Exclude individual passing-test lines from the model-facing response, but keep the pass count.
4. Build a stable schema. Use predictable fields for test name, file, line, failure type, assertion, and relevant expected/actual values. Stable output is easier for the model and for your monitoring to consume.
5. Set sensible limits. Cap captured stdout and traceback length. Preserve the first relevant application frame plus a short context window, and provide a way to retrieve the full log if needed.
6. Test the filter itself. Include cases for multiple failures, captured output, setup errors, skipped tests, malformed reports, and a completely passing run.

A simple rule for output selection is: include every failure, summarize every pass, and add more context only when the first report isn't enough. If a run has 500 passing tests and two failures, your default response shouldn't include 500 success lines. It should report the pass count and the two failures with enough detail to act.

Keep the agent's ability to investigate. If the summary says a failure happened in a fixture but omits the captured output that explains it, the agent should be able to request the full trace or rerun that test with more verbosity. Without that "escape hatch," filtering can save tokens while making debugging slower.

Finally, measure the result in your own workflow. Track response size, input tokens, model calls, task completion, and any failures introduced by missing context. A filter that reduces tokens but forces more retries isn't a win. Compare like with like: same tasks, model settings, test suite, and verification standard.

When filtering can go wrong

Filtering is a trade-off, not a universal command to make every tool response tiny. Some failures only make sense with context a generic parser discards. Integration tests may print the only useful clue to stderr. Snapshot mismatches can require a meaningful diff. Setup or collection errors may not fit cleanly into an assertion-focused schema.

One common mistake is treating all non-failure output as noise. A passing test can still be informative when an agent is debugging a flaky test, a race condition, or a regression that only shows up in a certain environment. In those cases, your tool should support targeted verbosity or a rerun of the specific test - not flood every normal turn with full output, but also don't throw away the one line that matters.

A second risk is fragile parsing. Regular expressions tuned to one pytest version or output format can quietly miss a new failure type. If you parse terminal text, "fail open": when parsing fails or the format looks unfamiliar, return a warning and enough original output to diagnose the run. Don't turn an uncertain parse into a misleading "all passed" summary.

Also think about security. Test output can include credentials, personal data, or environment details. Less output reduces exposure, but a filter isn't automatically a redaction system unless it explicitly detects and masks sensitive values. Keep secrets out of test output in the first place, and redact before logs reach an LLM service.

Use the filter when test output is a recurring source of prompt growth and your parser can preserve the information needed to act. Keep full logs accessible for the hard cases. For a short local script with one small test file, the engineering cost of a dedicated filter might not be worth it. In a long-running agent loop, a repeatable wrapper usually pays off because the same output policy applies across many runs.

Conclusion

Raw test output is often the wrong default payload for an AI coding agent. It repeats passing results, carries terminal noise across turns, and can bury the failures that matter. A deterministic filter can convert that stream into a compact report while keeping full logs available for deeper diagnosis.

In the benchmark, structured filtering reported a 94.1% reduction in cumulative input tokens with the same 20/20 hidden-test score for both modes. Treat that as evidence to run your own comparison, not as a promise for every codebase. Start by structuring failures, summarizing passes, preserving a path to full diagnostics, and measuring whether the agent still fixes and verifies work correctly.