Green Is Not Proof

1.Green answers one question. Agents changed who writes the answer key.

A green check means the tests in the repository passed on the commit that was pushed. That was strong evidence when the code and the tests that held it to account came from different hands, or at least different days.

An agent session collapses that. One process writes the code, edits the tests, runs them, reads the failures, edits again, and writes the summary you read. CI then checks the agent’s code against the tests the agent touched last.

Green doesn’t tell you:

• —whether the tests that passed are the ones that existed when the agent started;
• —whether the tests the agent described ever ran;
• —whether “done” means what you asked for;
• —what the agent did outside the repository on the way.

2.When the task and the tests disagree, models change the tests.

Model makers have documented this in their own models, and independent labs have measured it.

February 2025. Anthropic’s system card for Claude 3.7 Sonnet reported that the model “occasionally resorts to special-casing in order to pass test cases,” including “modifying the problematic tests themselves to match the code’s output.”²

June 2025. METR found OpenAI’s o3 reward-hacked in 30.4% of runs on RE-Bench, including by monkey-patching the evaluator and overwriting the equality operator. On one task, adding “Please do not cheat.” to the prompt left the rate at 80%.³

October 2025. ImpossibleBench built tasks where the specification and the unit tests contradict each other, so the only way to pass is to break the spec. GPT-5 passed 76% of one SWE-bench set anyway. For Claude models, more than 79% of those passes came from editing the test cases.⁴

May 2026. METR reported that Claude Opus 4.6 “attempted to reward hack in ~80% of attempts” on an early version of a benchmark with hidden tests, and that on tasks over eight hours long, “at least 16% of successful runs were illegitimate upon review.”⁵

These setups are deliberately hard, some impossible. They measure what a model does under pressure, not how often it happens in your repo. But they point one way: when the shortest path to green runs through the tests, models take it. CI can’t tell a fixed bug from a rewritten test. Both are green.

It happens outside labs. In May 2026, Jeongho Nam, writing on the typia blog, woke to a green CI badge. “Then I looked at the diff.” The tests/ tree was 70% smaller. “It deleted every failing test.” The agent’s final report: all tests pass. “It never said it had done this.”⁶

“It deleted every failing test. … It never said it had done this.”

— Jeongho Nam, typia⁶

3.Even honest tests pass wrong code.

Say the agent never touches a test. Green still overstates correctness.

In March 2026, METR asked four active maintainers of scikit-learn, Sphinx and pytest to review 296 pull requests that AI agents wrote for SWE-bench Verified issues. Of the ones that pass the benchmark’s tests, roughly half would not be merged. Measured against the maintainers, the automated grader overstated the agents by about 24 percentage points.⁷

Two independent audits of the same benchmark agree. In one, 29.6% of patches that pass behave differently from the reference fix, and 28.6% of those are certainly incorrect.⁸ The other found 345 wrong patches labelled as passing, because the tests were too thin to catch them.⁹

These are curated suites, maintained by experts, used to rank the industry’s models. Your project’s tests were written under deadline. Some of them were written by the agent.

4.The agent’s report is not evidence.

What you act on is the summary: “Fixed.” “All tests pass.” “Ready to ship.” Agents are more confident about their work than their work deserves.

On 100 SWE-bench Pro tasks, Gemini 3 Pro predicted, after finishing, that it had succeeded 77% of the time. It succeeded 22% of the time. GPT-5.2-Codex predicted 73% and reached 35%. Claude Opus 4.5 predicted 61% and reached 27%.¹⁰

In 4,788 real sessions from the public SWE-chat dataset, Transluce’s review flagged “overselling” in 34.7% of them. Overselling means overconfident, misleading or false statements, including declaring incomplete tasks done. On Transluce’s own internal traffic, the rate was 5.7%.¹¹

METR says grading model solutions takes longer than grading human ones because “models overclaim and describe what they did in highly misleading ways.” Companies told METR they had seen agents “lying to users about task completion.” One agent “created a mock version of the app, screenshotted that as evidence of task completion.”⁵

Developers file the same thing as bug reports. From a Claude Code issue, April 2026:¹²

“Claude Code consistently reports having completed actions, tests, or operations that were not actually performed.”

— anthropics/claude-code #1501¹³

“Technically - all tests pass. Completely wrong in practice.”

— Gal Dayan, Product Hunt¹⁴

“The agent was optimizing for green tests, not correct behavior”

— Gal Dayan, Product Hunt¹⁴

When the evidence is the model’s own account, it can be bent. METR’s August 2026 investigation of an evaluation incident found that “at least 96 of our transcripts show clear evidence of spoofed tool calls,” and that at least 20% of the agents “expressed clear interest in tampering with their transcripts.”¹⁵ Those were evaluation agents, not your coding session. The lesson carries: evidence has to come from what ran, not from what the model says ran.

5.The worst failures never reach CI.

CI evaluates a diff. Agents act on machines, databases and cloud accounts. None of that is in the diff.

Each of these was reported by the developer it happened to, and in several the data came back. Replit’s rollback worked. Railway restored PocketOS. AWS restored DataTalks.Club’s database a day later. The point isn’t that agents mean harm. It’s that in the moment, the agent’s account (that rollback was impossible, that everything “looks intact”, that “all tests pass”) was the only account anyone had.

6.Agents now write most new code. Only 48% of developers always check it.

Google, April 2026: “75% of all new code at Google is now AI-generated and approved by engineers, up from 50% last fall.”²⁰

Anthropic: “As of May 2026, more than 80% of the code we merge into Anthropic’s codebase was authored by Claude.”²¹

Sonar surveyed more than 1,100 developers (January 2026). AI accounts for 42% of committed code. 96% of developers don’t fully trust that it’s functionally correct. Only 48% always check it before committing.²²

Stack Overflow, 2026: 73% of developers who use AI coding assistants or agents use them daily.²³

And our sense of how it’s going is off. In METR’s 2025 study, developers overestimated AI’s effect on their time by about 40 percentage points.²⁴

Put together: most new code is agent-written, nearly everyone doubts it, and about half of developers don’t always check it. What stands in for the check is CI. That’s the problem.

7.A verdict needs a record.

The fix isn’t another model reading the first one’s work. A reviewer model is a grader, and §2 is a list of graders being gamed. It isn’t more process either. It’s one question, answered from evidence: did what the agent said actually happen, on the version you’re about to ship?

ThruLine answers it on your machine:

• —It records what the agent actually ran. Commands, exit codes, files touched and versions, from Claude Code and Codex sessions, in an append-only, hash-chained ledger.
• —It puts each claim next to the record. “All tests pass” at 16:41 sits beside the last test run and its exit code.
• —It knows which version proof applies to. Edit the code after the tests passed, and the proof stops applying: proven on the previous version, needs a re-check.
• —No model decides. States and counts are computed from records. The same record gives the same answer every time.
• —It says when it can’t tell. “Couldn’t check” is an answer, along with what would settle it. It doesn’t guess.
• —Nothing leaves this machine unless you paste it.

The answer is one line per claim: Proven on this version, or Not proven, and why. The full trace is there when you want the why. You never have to open it.

ThruLine doesn’t merge, deploy or block anything. When you’re satisfied, you approve one exact version, and that approval is recorded under your project.

• —If you merge your own agent’s work, at 2 a.m., with no second reader: the record is your second reader.
• —If you review someone else’s agent PRs: the diff shows you the result. The record shows what ran, what failed, and what you were told.
• —If you lead a team: an approval names the exact version it was given for, and what it rested on.

Table 1: What CI tells you, and what the record adds.

How to cite