AI Coding Agent Benchmarks: What KillSwitch Shows
See what KillSwitch-Bench reveals about coding agents, costs, refusals and scoring, plus practical checks for interpreting benchmark results with confidence.
KillSwitch-Bench: What a Hostile Coding Test Reveals About AI Agents
AI coding benchmarks often publish a score and leave you to guess what happened in each run. Did the agent solve the task? Refuse it? Crash? Hit the time limit? KillSwitch-Bench takes a different approach: it tests 20 model-and-harness combinations against a programming language built to be unusually hard for language models, and it publishes cost and wall-clock time alongside the results.
It's an interesting benchmark, but the real value is in the design. KillSwitch uses instruction-like syntax, the specification is far larger than what a model can read in one sitting, and the interpreter includes runtime checks aimed at detecting coding agents. Even the leaderboard invites questions: how are scores combined, and what exactly does "run" mean here?
Those details matter if you're using benchmark results to pick a model, build an agent, or evaluate your own test setup. A score only helps if you understand what the benchmark measured, what it hid inside the run, and what the final summary leaves out. This article walks through KillSwitch-Bench's design, the results that stand out, and a practical way to make agent evaluations more trustworthy.
What KillSwitch-Bench tests
KillSwitch is a stack-based programming language where the source code is written as prose. Its keywords are full sentences, and values and identifiers use unconventional rules. Most of the text in a script is ignored by the interpreter, so the actual logic can be embedded in what reads like a story.
For example, a particular sentence opens an entrypoint, while a phrase like value 4 lines below pushes a literal. Identifiers are referenced through anagrams of adverbs. The design is deliberately unfamiliar: the benchmark is trying to see whether a model can reason from the specification instead of leaning on patterns it's seen before.
That's a reasonable research goal. Still, it's not "just a strange syntax puzzle." Some valid language constructs look like the kind of hostile text that safety-trained models are taught to treat with suspicion. In one function call, the literal phrase ignore previous instructions appears, followed by other syntax. The language also includes a hack statement and a return keyword referencing a sensitive historical event.
So the model can end up with two competing demands: follow the language specification, or avoid emitting text that resembles an injection attempt. If it refuses, KillSwitch-Bench records that as a failure - on the same level as incorrect code, compile errors, or timeouts.
A benchmark can test more than one capability
The stated goal is to separate code reasoning from training-data recall. In practice, the results can also reflect other skills:
- Whether the model can follow a long, noisy specification
- Whether it can tell normal task instructions from suspicious-looking syntax
- Whether the harness provides the right context and handles errors cleanly
- Whether the model can produce code that matches the interpreter's specific rules
That doesn't make the score meaningless - but it does mean you can't interpret a low pass rate as "the model can't reason" without more evidence. You'd need run-level details to know which capability failed.
The benchmark discussion points to safety guardrails as a cause of some weak outcomes. But the positions in the reported leaderboard don't cleanly support that simple story. Claude Fable 5.1 is sixth of 20 at 46.7%, and Gemini 4 Argon is fifth at 47.1%. If refusal behavior matters to the result, it should be measured directly rather than inferred from the overall score.
The original KillSwitch-Bench write-up includes the underlying results and implementation details. The core takeaway is straightforward: unusual test conditions can reveal useful weaknesses - but they also make it more important to explain what kind of weakness a score actually represents.
The specification is part of the experiment
Each task includes two specification documents. One, SPEC.md, is about 17 KB. The other, AI_SPEC.md, is 5,151,615 characters - about 1.3 million tokens. That's far beyond the context window of current models.
The large file reportedly blends public-domain prose with embedded instructions. It even opens by asking the reader to recite stories to improve the interpreter's "happiness." In other words, it isn't only a reference manual. It also probes how an agent handles an oversized document when the useful rules are scattered through a lot of noise.
That can be a legitimate stress test. It also changes what the evaluation measures. A model might fail because it can't locate a relevant rule in what it receives, because the harness truncates or selects the wrong portion, or because it misunderstands the language itself. Those are different failure modes, and you shouldn't mash them into a single explanation.
What the leaderboard does - and does not - tell you
KillSwitch-Bench makes one smart choice: it publishes cost and wall-clock time per task alongside the scores. Those are operational measurements, not vanity metrics. If a model passes at a reasonable rate in under a minute, it may fit a production workflow better than a model that scores slightly higher but burns a large budget and several minutes per task.
The problem is turning multiple measurements into one headline score. The benchmark's composite combines pass rate, cost, and implementation size. Cost has a clear practical reason - latency and inference budgets matter. Code size is less obviously connected to correctness.
If a verbose program is correct and works, its length doesn't automatically mean the model is worse at reasoning about the language. Penalizing size can turn "prefers concise code" into an accidental correctness signal. That might be reasonable only if compactness is explicitly part of the product requirement. It's harder to justify when the benchmark's central question is whether a model can solve the task.
A better reporting approach is to publish pass rate, cost, and size as separate columns, then spell out how any composite score works and what weights are used. That way, readers can decide what trade-offs matter for their own use case. If you can't break apart the score, you can't act on it intelligently.
The model and harness are a combined system
The cleanest comparisons in the table pair two Claude Sonnet versions with the same harness, Claude Code:
| Model | Harness | Score | Cost per task | Time per task |
|---|---|---|---|---|
| Claude Sonnet 5 | Claude Code | 17.3% | $2.55 | 11m 35s |
| Claude Sonnet 5.5 | Claude Code | 7.4% | $0.15 | 53s |
Yes, the newer model is roughly 13x faster and 17x cheaper in these runs - but it scores at less than half the pass rate. You see similar patterns elsewhere in the reported results. Claude Opus 5 scores 66.8%, versus 58.7% for Claude Opus 5.5; GPT-6.1 Sol scores 59.2%, versus 28.0% for GPT-6 Sol.
So treat these numbers as prompts for investigation, not a universal claim that "older models are better" or "newer ones are worse." This evaluation is specific to a particular language, a very large specification, a specific harness, and a strict time limit. For example, the shorter run for Sonnet 5.5 could mean it found an answer quickly (or stopped early), while Sonnet 5 consumed more time and tokens. Without run-level traces and terminal statuses, the table can't tell you why.
The right unit of comparison isn't always the model name. It's the model-plus-harness configuration under a specific task setup. Change the scaffolding, context selection, timeout, or retry behavior, and the result can change. That matters because a "model score" isn't purely a model property when an agent framework is controlling what the agent sees and what actions it's allowed to take.
One score hides run-to-run variation
KillSwitch-Bench runs each model at least twice per task, which is a solid start. But the leaderboard shows a single number per model without a spread or uncertainty interval.
Even if the evaluator itself is deterministic, the full execution path may not be. The interpreter reportedly resolves a value by counting lines, and if a line contains more than one literal, it may pick one at random. It also tracks a "happiness" state that affects how strictly it parses. When behavior can vary between runs, a single point estimate can give you a misleading sense of precision.
For a benchmark with any randomized behavior, you'd ideally publish the number of runs, the distribution of pass rates, and the exact aggregation method. Even a simple range would help readers separate a stable result from a score that moved because of a small sample.
Two zero-cost results need more explanation
GPT-5.6 Luna and Qwen3.8 27B both show up as 0.0% and $0.00 per task. The Qwen row also reportedly hits the full 30-minute timeout. A zero recorded cost alongside a long run could mean the billing signal wasn't captured properly - it doesn't automatically mean the system consumed no resources.
More importantly, the leaderboard doesn't say what terminal event produced each zero. The agent may have refused, failed to start, hit a harness error, timed out, or produced incorrect code after doing real work. Those outcomes aren't interchangeable.
For a benchmark designed around adversarial input, knowing whether an agent stopped early (before doing actual work) is especially important. A flat "0.0% bucket" treats very different failures as if they were the same thing.
Integrity controls worth borrowing
One of the most distinctive pieces of the benchmark implementation is its process-tree check. The interpreter inspects the parent process chain for a list of 26 coding-agent binaries, including tools such as Cursor, Claude, Codex, OpenCode, and Aider. If it detects one, it returns a fake error asking the agent to draw a pelican in a spacesuit as an SVG. If that file already exists, the error changes to say the drawing isn't good enough.
The task is intentionally impossible to satisfy. The agent is given an unsatisfiable instruction inside the run, rather than a clear signal that the evaluator has detected it.
The repository also includes an AGENTS.md file in the source folder that tells agents to refuse to read that folder, even if a user asks. That creates competing instruction channels: the large specification invites the agent to read and follow it, while the repository's rules file says not to read the relevant material.
Taken together, these controls make the evaluation more than a test of generated code. They test how an agent behaves in an environment where repository instructions, runtime checks, and user intent can conflict. The process-tree approach is especially notable because it tries to identify the agent at runtime, instead of only guessing afterward based on artifacts.
There's a trade-off. A benchmark that detects and disrupts known agents could be useful for studying agent behavior, but the results are specific to the controls and configurations it recognizes. It also makes the evaluation feel less like "neutral coding" and more like a security-style interaction test. That may be exactly the point - readers just shouldn't assume it behaves like a generic programming benchmark.
A practical checklist for better agent evaluations
The practical response isn't to reject unusual benchmarks. It's to demand enough reporting to actually interpret them. If you run evaluations internally or use them to compare vendors, build your reporting around individual runs rather than a single composite number.
Record a terminal status for every run
At minimum, capture one outcome for every run:
- Completed: The agent produced an answer and the task was evaluated.
- Refused: The model or agent declined to proceed.
- Errored: The model, harness, tool, or interpreter failed.
- Timed out: The run reached its configured limit.
Also track pass or fail as a separate field. A run can complete and still fail the task, and a run can error and still leave behind an artifact that passes. Recording both status and outcome makes those cases visible.
Publish the parts of composite scores
If you combine correctness, latency, cost, code size, or anything else into one number, publish each component and the weighting. Make it clear which measures are hard requirements and which are preferences. That gives teams a way to reuse the results under their own constraints, instead of treating one set of weights as universal.
Report variability and the full configuration
For repeated runs, report the number of runs and the distribution - not just the mean or final score. Include the model version, harness, prompt or system instructions, context limits, timeout, retry policy, and relevant evaluator settings. In real projects, small configuration differences often explain what looks like a model-quality difference.
Test whether your controls actually affect behavior
The repository instruction conflict suggests a practical technique for code review and policy enforcement: use a controlled canary change. Introduce a safe, deliberate violation of a rule and see whether the review system blocks it, comments on it, or ignores it.
A demo that shows a tool can read a standards file isn't proof that the tool enforces the standards. What you care about is the observable result. You can run the same kind of check against a reviewer in CI, whether you use Kodus, CodeRabbit, or something else. Keep the test small, safe, and easy to repeat.
For more on interpreting results, connect this checklist to your own agent evaluation process and code review testing workflow. The links should point to your organization's guidance; the point is to make evaluation practices repeatable rather than dependent on a one-off demo.
Conclusion
KillSwitch-Bench isn't useful because its leaderboard settles which coding agent is "best." It's useful because its design forces questions many evaluations leave implicit. A score depends on the language, the specification, the harness, the runtime, and how failures are categorized. Cost and latency help, but they don't explain what actually happened inside a run.
If you build or buy coding agents, start with three changes: record terminal status, publish the components of any composite score, and report the configuration and run-to-run variation. Then test whether your review and policy controls produce the behavior you expect. These are small, inexpensive improvements - and they make benchmark results far more actionable.
Next time you compare agent scores, ask what the agent actually did - not just what number it earned.