Your AI agent says “tests pass”. 1 in 5 times, there’s no proof.

TL;DR

We are publishing an open benchmark for one question: when an AI coding agent says “tests pass”, does the record of what actually ran agree? It has three parts: a planted-lie suite of 9 lies and 15 honest twins, a labelled set of 640 sentences from 150 real agent sessions with a sealed test half, and a held-out set of 25 claims the tools have never seen.

• The problem is real and measured. In our labelled sample, 22% of check claims had no proof behind them (95% interval 18–27%). An independent study of 101 claims found 35% untrue when made. In both, most untrue claims were stale passes: tests ran, then the agent changed code.
• First results, our own tool included.
  • claimproof 0.15.0 caught 7 of 9 planted lies, but wrongly blocked 10 of 15 honest turns. Awaiting its maintainer’s reply.
  • ThruLine 0.2.0, installed as its README said, caught 0 of 9.
  • The current ThruLine v0.3 build marks 8 of 9 not proven and gives 8 of 9 a working receipt. It catches 3 of 9 live in the session, and 6 of 9 on a test build that adds the in-session warning.
  • Out of sample, ThruLine’s claim finder missed 7 of 25 lies and false-alarmed on 5 of 25 honest sentences.
• What we have not measured yet: any other tool. We will only publish a tool’s numbers after running its pinned version under a preregistered protocol and giving its maintainers a chance to check our setup.
• Small numbers, honest intervals. Nine planted lies is a smoke test, not a verdict. v1.0 aims for 300+ cases, most of them contributed by people who are not us.

Why a benchmark, and why now

AI review tools read the diff. Almost none check what actually ran. One 2026 comparison says the three best-funded reviewers “generate their review by reasoning over a diff… none of them actually executes the code” (Tech Insider). Meanwhile the agents writing that code end their turns with “all tests pass”, and between a fifth and a third of the time it isn’t true when they say it.

Tools to catch this appeared in 2026: side-channel recorders, re-run harnesses, Stop-hook gates, LLM judges. There is no shared scoreboard, so each publishes its own anecdote, or nothing. Research on the related “false success” problem found LLM judges at AUROC 0.54–0.65, below simple text detectors (survey). Without a common test, nobody can tell which approach works.

This benchmark is that test. We built ThruLine, so we have a conflict of interest. The section How we keep ourselves honest says how we handle it.

What 357 real claims looked like

Half of the time an agent said a check passed, a real passing run backed it. One in five times there was nothing behind it, and nearly all of those were stale passes: the tests had passed, and then the agent changed more code.

Two things follow. First, agents rarely lie outright: only 5 of 357 claims were contradicted by the record. The problem is timing, which a text-only checker can’t see. Second, three in four test runs went through tail or head, so the transcript the agent leaves behind often doesn’t hold the result at all. A tool that reads transcripts is reading a summary the agent chose.

What’s in the benchmark

It has three parts. Each answers a different question, and none of them alone is enough.

1. Planted-lie suite K: can a tool catch the classic lies without crying wolf?

Each case is a fresh copy of a small Node app (tinylink, 18 tests). It gets one agent turn with real file edits and real shell runs, then a final reply. Every lie has an honest twin that looks almost the same, so a tool that flags everything scores badly.

That’s 9 lies and 15 honest cases. A correct tool flags all 9 and none of the 15.

2. Real-session set GTT-1: how often are real claims backed?

• 640 candidate sentences, a stratified sample from 150 of our own Claude Code sessions, split by session lineage. DEV has 310 rows for development. TEST has 320 rows, sealed: its labels were hashed (sha256) before any code change, and it is opened once per tool version, for the final run.
• Labels come from a written codebook, checked against the record, never against the agent’s words. For 118 rows there is also an independent oracle: gate logs or re-runs.
• Agreement before any codebook change (Cohen’s kappa, 120 rows labelled twice, blind): 0.966 on “is this a check claim”, 0.816 on the 6-class verdict, 0.686 on verdict plus reason.
• Privacy: these are our own sessions. We release the codebook, the labels and each claim sentence after a redaction pass. Any row we can’t redact cleanly is dropped, and we say how many.

3. Held-out claim set: does the tool generalise?

25 lies and 25 honest sentences that no tool author tuned against. A reviewer who doesn’t write the claim finder keeps it. A score here is the closest thing to “what you’ll see in your own sessions”.

What we measure

The headline number is how many lies a tool catches with how few false alarms, and one failure outranks both.

Rules for counting:

• A block is the documented block decision on the Stop event. A flag is a visible in-session line that names the claim. Both count as live catches, reported separately.
• Every rate comes with its count and a 95% Wilson interval. On 9 lies, “7 of 9” has an interval of roughly 45–94%. We print that, and we don’t round small samples into big claims.
• Each tool runs at a pinned version (package hash in the report), installed the way its own README says, in a scratch home that never touches a real user’s files.
• If a tool offers a stricter mode, it’s reported as a second configuration. The README route decides the headline.

Results so far

No tool we measured is good enough yet, ours included. The one that catches the most lies also blocks two in three honest turns, and the one that never false-alarms catches only a third of the lies live.

Awaiting reply. claimproof’s numbers below are awaiting its maintainer’s reply. Under our right-of-reply rule they have seven days to point out a misconfiguration. If they do, we’ll rerun, update this post and say that we did.

Planted-lie suite K, case by case (✓ = flagged, · = missed)

What the table says:

• claimproof reads only the agent’s final words and the lines near them. That’s why it catches almost everything and also blocks honest turns: it can’t tell a real run from a claimed one. It has no receipts and no record view.
• ThruLine 0.2.0 caught nothing because its default install didn’t record test runs. We published that on day 1.
• ThruLine v0.3 merged catches lies in its record and receipts (8 of 9). Its live catches are 3 of 9 until the in-session warning lands. The test build with that warning reached 6 of 9 live with 1 false alarm. K2 is invisible to every tool here: no run, and the words look like a real result.

Who we measured, and who’s next. We compare approaches, not logos. claimproof isn’t a funded rival. It’s a small open-source package, and we measured it because it was the only other tool that installs as a Claude Code Stop hook and runs locally, so it’s the cleanest example of the “read the agent’s words” approach. The next round adds the approaches people actually use:

The last row is the one we most want to see. It tests whether a reviewer reading the code notices that the claim about running it is false.

Held-out claim set (25 lies, 25 honest), ThruLine’s claim finder only: 18 of 25 lies found (72%, interval 52–86%), 5 of 25 honest sentences wrongly taken as claims (20%, interval 9–39%). A missed claim reads “couldn’t be checked”, never Proven, but this is below our own bar. We say so in the v0.3 release notes and fix it in v0.3.1.

False Proven: known cases open in ThruLine, found by our own review (among them a narrowed run read as a full one, and a .. path case). All of them block the v0.3 release, and no build ships with any of them open.

Real-session set, by tool: not yet run on TEST. We’ll run each tool once per version and publish its result, whatever it is.

How we keep ourselves honest

We build one of the tools being scored, so the rules below exist to stop us grading our own homework.

• Preregistered. The question, the unit (one agent turn), each tool’s configuration, the block test and the strata were written down and hashed before any tool ran on real sessions. Changes are logged as numbered deviations, never edited in place.
• Pinned versions. Every tool runs from a hashed package (claimproof 0.15.0 wheel 0a453b44…, ThruLine 0.2.0 binary c0488ced…), installed by its own README.
• A sealed test half. TEST’s labels were hashed before any ThruLine code change. We tune on DEV and the planted suite only. TEST opens once per tool version, and that run is the one we publish.
• Every case is public. The builder script, the fixture app, every transcript and each tool’s raw output per case. Anyone can rerun them and get the same table.
• Right of reply. Before we publish a tool’s numbers, its maintainers get our exact setup and seven days to point out a misconfiguration. If they’re right, we rerun, and we say that we did.
• We publish our losses first. Our own 0 of 9, our false-alarm rate and our open false-Proven cases are in this report, above any claim about anyone else.
• What would change our mind. If an LLM judge or a diff reviewer beats record-based tools on the sealed half, we’ll publish that too.

Run it yourself, or put your tool on the board

The suite, the harness and every raw result go public with ThruLine v0.3. The data is under CC BY 4.0, the harness under MIT.

To reproduce:

1. Build the cases: one script makes each case from the fixture app, with real edits and real runs, on macOS or Linux, with Node 22 and Python 3 (standard library only).
2. Install the tool under test into the scratch home the harness creates, using the tool’s own README.
3. Run the harness. It replays each case through the agent’s real hook events and writes one row per case: flagged or not, how, and the tool’s raw output.

To submit a tool:

• What it can plug into. v0.1’s harness drives Claude Code Stop hooks. Tools that work another way (a re-run harness, an IDE extension, a PR reviewer) need a small adapter, and we’ll write it with you.
• What you send. A pinned version and an install command. We run it under the preregistered protocol and send you the results before anything is published.
• New cases are welcome, especially lies that fool ThruLine. A case is accepted when its lie and its honest twin both reproduce on a clean machine.

Where it goes next:

Limits, and what we’d like to be proven wrong about

• Small. 9 lies and 15 honest cases can rank tools roughly, but they can’t prove a winner. Intervals overlap. v1.0 needs hundreds of cases.
• Narrow. It covers claims about tests, builds, lint and typechecks only. “The feature works”, “I refactored safely” and work handed to sub-agents are out of scope. Sub-agent handoffs were 39 of the 51 claims nobody could check in our sample.
• One fixture, one language. The planted cases are a small Node app. Python, Rust and Go cases come in v0.2.
• Our sessions, our habits. The real-session set is 150 sessions from one team building one product. The independent 101-claim study agrees on the main pattern (stale passes dominate), but more people’s sessions would make it stronger.
• Labels by AI graders. Two models labelled the rows, with a third resolving disagreements, against a written codebook, and agreement is reported. Human relabelling of the sealed half is planned for v1.0.
• Built by an interested party. The rules in How we keep ourselves honest reduce the bias, but they don’t remove it. The real fix is the hidden test set held by someone else, in v1.0.

If your tool beats ours on this benchmark, we want to know first, and we’ll publish it.

Sources

• When a Coding Agent Says Done: a 2026 survey of the verification layer
• I checked 101 “tests pass” claims from my AI coding agents. 35% weren’t true.
• CodeRabbit vs Greptile vs Qodo: AI code review tools 2026
• ThruLine build log, day 1: ground truth first
• ThruLine build log