[ ThruLine research · The Done Map · October 2026 ]

Green tells you what you didn’t break.

We mapped 701 times an AI coding agent said it was done: every command it ran first, the last test it saw, and what the hidden tests said. Its final test run was green 85% of the time, on easy tasks and impossible ones alike. On 25 tasks that none of 134 other systems has solved, it was green 22 times and right 3.

Ashwinsuriya Ravi · Founder·17 October 2026·15 min read·public SWE-bench runs, 33,114 agent steps

Agent’s last test 0 greenHidden tests 0 of those wrong

701

times the agent said it was done, on 800 tasks in 53 repositories and 10 languages

80.1%

were right. The hidden tests passed on 560 of 699

596

ended on a green test run. Right 79.5% of the time. Runs that didn’t end green: 83.5%

3% · 19%

of green runs broke old behaviour · missed the new one

22 of 25

runs ended green on tasks none of 134 other systems solved. Right on 3

Why we did thisAn agent that says “done” knows something. We wanted to know what.

When a coding agent finishes, it has a record: the files it changed, the tests it ran, the output it read. That record is all the agent has when it decides to stop, and it is usually all you have when you decide to trust it.

So we took a set of runs where the truth is known. SWE-bench gives an agent a real issue from a real open-source project and grades the fix with tests the agent never sees: new or changed tests from the pull request that actually fixed it. Its leaderboard publishes some systems’ full runs. We used the most recent: Gemini 3.5 Flash in the open mini-SWE-agent harness, on SWE-bench Verified (500 Python tasks) and SWE-bench Multilingual (300 tasks in C, C++, Go, Java, JavaScript and TypeScript, PHP, Ruby and Rust).¹

For every run we read every step, found the moment the agent submitted, and asked what its record said at that moment. Then we put that next to the hidden tests’ verdict. The agent tested a lot: in 94% of runs it ran tests after its last edit, a median of 4 test runs per task. The question is what those runs were telling it.

Figure 1The Done Map

Each square is one task. Start with what the agent’s own last test run said before it submitted, then switch to what the hidden tests said. The squares that turn orange are the ones the record didn’t warn you about.

Figure 1. 800 tasks, grouped by language and repository. An outlined square has no published run, a run that never submitted, or no evaluation report. Hover or tap a square for its record.

SWE-bench experiments, entries 20260901_mini-v2.4.2_gemini-3-5-flash and 20260902_mini-v2.4.6_gemini-3-5-flash · trajectories and evaluation reports

By language and repository

Tap a language to see its repositories. Shading is darker where the share is higher.

Table 1. “Done” counts runs that submitted and have a hidden-test verdict. “Missed the new behaviour” means at least one of the task’s target tests failed. “Broke old behaviour” means at least one test that used to pass now fails. A run can do both. “Said done when it submitted” counts runs with at least one such sentence at or before the first submit.

Finding 1A green last run didn’t mean a better fix.

596 runs ended on a green test run: the agent edited, ran tests, they passed, and it submitted. Those runs were right 79.5% of the time. The 63 runs that ended on a red run were right 85.7% of the time. Neither number is far from the overall 80.1%.

If you had used the agent’s final test run to decide which fixes to trust, you would have trusted the green ones and been wrong about 122 of them. And the 40 runs that never re-ran tests after their last edit? Right 80.0% of the time.

Share of runs the hidden tests passed, by how the run ended

Dot is the share; line is the 95% interval. The dashed line is all 699 runs.

Figure 2. “Ended green” means the last test run after the agent’s last edit passed. “Edited after a green run” means a run passed, then the agent edited again and didn’t re-run. Wilson 95% intervals.

Finding 2Green tells you what you didn’t break.

Look at how the 139 wrong runs were wrong. In 130, at least one of the task’s target tests failed: the new behaviour the issue asked for wasn’t there. Only 21 broke something that used to work. Among green runs the split is starker: 19.0% missed the new behaviour, 3.0% broke the old.

That is the whole story in one comparison. The agent’s test runs were good at the question they could answer, did I break anything that was already tested? They could barely touch the question that decides “right”, does the new behaviour work?

They couldn’t, by construction. The tests that grade the new behaviour come from the fix’s own pull request, and SWE-bench keeps them hidden. The harness also told the agent, in capitals, DO NOT MODIFY: Tests. Its suggested workflow was to write a script that reproduces the issue and run it again after the fix. So a green run checked the old behaviour with the project’s tests, and the new behaviour with the agent’s own reading of the issue.

What went wrong in the 139 wrong runs

Each block is one run.

Figure 3. Target tests are SWE-bench’s FAIL_TO_PASS tests; old behaviour is PASS_TO_PASS. Two runs’ patches did not apply and count as missing the target.

One run, start to finish

On hashicorp__terraform-34900 the agent edited two files, then ran go test ./internal/terraform and go test ./internal/configs/configschema/.... Both passed. It submitted. The harness didn’t register the submit, and when the agent sent it again it added: “I have completed the task and verified that all tests pass.” That sentence was true. The hidden tests then ran the two target tests that came with the real fix, TestContext2Apply_sensitiveNestedComputedAttributes and TestContext2Plan_nestedSensitiveMarks. Neither name appears anywhere in the agent’s run. Both failed.

Finding 3The hardest tasks were the greenest.

SWE-bench Verified publishes per-task results for many systems. We took the 134 other entries that list which tasks they solved (October 2023 to December 2025) and sorted the 500 Verified tasks by how many of them solved each one.

The agent’s chance of being right falls exactly as you’d expect: 99.3% on tasks three quarters of systems solve, 12% on tasks none of them did. Its chance of ending green does not move. It sits between 81% and 89% at every level. 32 Verified tasks have never been solved by any of the other systems, and 25 of them have a run from this agent with a verdict. On 22 it ran its tests, watched them pass, and submitted. It was right on 3.

Ended green vs hidden tests passed, by task difficulty

SWE-bench Verified · easiest tasks on the left · difficulty is the share of 134 other leaderboard entries that solved the task

All 500 Verified tasks, easiest on the left

Bar height: share of other systems that solved it. Colour: how this agent’s run ended.

Figure 4. Top: two series across five difficulty bands, easiest first, with 95% intervals. Bottom: one column per task. Orange columns are runs that ended green and were wrong. 58 of those 76 sit in the two hardest bands, which hold a quarter of the tasks.

Finding 4What it said didn’t help either.

This agent narrates plans, not results. Of the 27,648 messages in which it explained a step, 98.7% begin “I will” or “I’ll”. When it submitted, it said the work was finished, fixed or verified in only 40 runs, 44 sentences in all. Those runs were right 67.5% of the time (95% interval 52.0 to 79.9). The other 659 were right 80.9%. Forty runs is a small sample, so read that as “no better”, not “worse”.

Count every sentence in the logs and you get a much bigger number, 162, for a reason that has nothing to do with the agent. In 138 Multilingual runs the harness didn’t register the first submit, almost always because a shell warning was printed ahead of the submit marker. The run carried on, and the agent submitted again. Nothing was edited or tested in between. In 80 of those runs the second try came with a sentence saying it was done: 118 sentences. We keep them out of the numbers above, because they were written after the decision to stop. They still describe the same work, and they read the same.

Here is what it said in some of the runs the hidden tests failed, each marked with when it was said. Several are true as written. The tests it ran did pass.

Finding 5Red didn’t mean wrong.

63 runs ended on a failing test run, and the agent submitted anyway. 54 of them were right. In 9, the exact same command was already failing before the agent changed a line. A red run in someone else’s suite is often old news.

Exit codes need care too. On projectlombok__lombok-2792, the build tool printed BUILD SUCCESSFUL and exited 0 while its own summary said Tests run: 340, Failures: 1. If you read only the exit code, that run was green.

Figure 5Every step of 699 runs

One row per run, one cell per step, left to right until it submitted. The rows are grouped by how the run ended and whether it was right. If the record could tell you which fixes were wrong, the orange group would look different from the one above it. It doesn’t.

Figure 5. Steps beyond 120 are cut off (2 runs). Each row ends where the agent submitted. Cell height also encodes the kind of step, so the chart reads without colour: edits and test runs are full height, scripts half, reading a sliver.

What this meansA record can prove the tests passed. It can’t prove what the tests were for.

None of this says the agent lied, or that testing is pointless. Its tests caught almost every regression: only 3% of green runs broke old behaviour. It says that a passing run proves the claim it actually supports, these tests passed on this code, and nothing bigger.

That limit applies to us too. ThruLine checks an agent’s claims against what it actually ran. Pointed at these 699 runs, it would have found a passing test run on the final code 596 times, correctly, including the 122 runs where the fix was wrong. The record cannot say whether the new behaviour works unless someone has said, before the work starts, what “works” means: the test, the example, the check that has to pass. That is why ThruLine asks for what counts as done and doesn’t treat a green run as approval. On these tasks, that check existed only inside the benchmark.

If you are reviewing an agent’s work, the practical version is short. Ask which test proves the new behaviour, and look for that test in the record. A green run without one tells you what didn’t break.

MethodHow we counted

Data

Two entries in the SWE-bench experiments repository, both Gemini 3.5 Flash in mini-SWE-agent: 20260901_mini-v2.4.2_gemini-3-5-flash on Verified (leaderboard score 71.8%) and 20260902_mini-v2.4.6_gemini-3-5-flash on Multilingual (67.0%). Each publishes the full trajectory and the evaluation report for every task it ran.¹ We cloned both on 17 October 2026 and analysed every file. These are the only recent entries whose trajectories are published in a git repository; older entries keep theirs in cloud storage we could not reach.

Of 800 tasks, 95 have no published run (59 on Verified, 33 of them in matplotlib; 36 on Multilingual), 4 runs ended without a submission the harness accepted (step limit or a server error; three of them had run the submit command), and 2 have no evaluation report. That leaves 699 runs with both a “done” and a verdict. The leaderboard scores count the missing runs as failures; our rates do not, which is why ours are higher.

Definitions

Done

The first time the agent ran the submit command, echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT. Its instructions say: “When you’ve completed your work, you MUST submit your changes.”

Edit

A command that writes to a file that appears in the final patch: sed -i, a heredoc, a scripted find-and-replace in Python, Node, Ruby or Perl.

Test run

A command that starts a test runner or build check: pytest, Django’s runtests.py, SymPy’s bin/test, go test, cargo test, Maven, Gradle, ant test, npm and yarn test scripts, Jest, Mocha, Karma, PHPUnit, RSpec, Ruby test files, make check, ctest, Redis runtest, Google Test binaries. Runner names inside a heredoc or a quoted script don’t count.

Green, red

Red if the output reports a failure or error count, a failed build, or the command exits non-zero. Green if it exits 0 and reports no failure. When output is piped or sent to a file, we need a pass summary to call it green. A runner that wasn’t installed, timed out, or collected no tests is neither.

How it ended

Ended green: the last test run after the last edit was green. Ended red: it was red. Edited after a green run: a run passed after an edit, then the agent edited again and didn’t re-run. Ran a script only, ran nothing: no test run after the last edit.

Right

SWE-bench’s verdict: every target test (FAIL_TO_PASS) and every existing test it checks (PASS_TO_PASS) passes.

Said it was done

A sentence saying the work is finished, fixed, verified or that tests pass. Plans (“I will…”) and negations don’t count. We count only sentences written at or before the first submit; the 118 written after it are reported separately (Finding 4).

Difficulty

The share of 134 other SWE-bench Verified entries, from October 2023 to December 2025, that list the task as resolved.

Checks

We read 50 random test runs by hand and agreed with the green, red or couldn’t-run label on all 50. That bounds the error rate at roughly 7% or less, not zero.
We read all 44 “done” sentences written at or before submit; each one says the work is finished, fixed or verified.
A second, independent implementation, written from the definitions on this page without our code, agreed with our ending label on 684 of 699 runs (97.9%). Under its labels, green runs were right 79.5% of the time and the rest 83.3%.
No run edited or tested anything after its first submit, so the record at the first submit is the record the hidden tests graded.
All 700 runs that changed files had at least one detected edit before they submitted.
Every rate comes with a Wilson 95% interval. Runs are the unit; we didn’t weight by repository.

LimitationsWhat this doesn’t show

One agent, one harness. Another model, or a harness that asks for a test of the new behaviour, could look different. That is the next map to draw.
Benchmark tasks. SWE-bench issues are public and may be in the model’s training data. The hidden tests are the benchmark’s definition of right, and they are not perfect.
Our parsing. Test and edit detection is pattern-based. The hand checks bound the error; they don’t remove it.
Two harness versions. Verified ran mini-SWE-agent 2.4.2, Multilingual 2.4.6. We report both together and separately in the data. In 138 Multilingual runs the 2.4.6 harness missed the first submit (Finding 4); we date “done” from that first submit.
Older comparison systems. The 134 entries behind the difficulty measure are from 2023 to 2025. This agent is newer and solved 3 tasks none of them did.

DataTake it with you

One row per task, 800 rows: benchmark, task, repository, language, whether a run was published, whether it said done, how it ended, whether the hidden tests passed, target tests passed of total, existing tests broken, steps, test runs, the “done” sentence it wrote when it submitted, how many it wrote after a missed submit, and how many other Verified systems solved the task.

done-map.csv ·

If you have agent runs with outcomes, from any model or harness, we’d like to put them on the map. Get in touch through thrulinehq.com.

A green run is a fact about the tests you ran. Whether you’re done depends on which tests those were.

NotesSources

SWE-bench experiments repository, github.com/SWE-bench/experiments, entries evaluation/verified/20260901_mini-v2.4.2_gemini-3-5-flash and evaluation/multilingual/20260902_mini-v2.4.6_gemini-3-5-flash. Trajectories and reports: john-b-yang/20260901_mini-v2.4.2_gemini-3-5-flash and SWE-bench/20260902_mini-v2.4.6_gemini-3-5-flash.
Difficulty: the results/results.json “resolved” list of every other Verified entry that publishes one, 134 entries.
SWE-bench: Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024. SWE-bench Verified is the 500-task subset reviewed by human annotators; SWE-bench Multilingual covers 300 tasks in 9 languages.
mini-SWE-agent instance template, quoted from the published trajectories: “DO NOT MODIFY: Tests”; “Create a script to reproduce the issue … Verify your fix works by running your script again”; “When you’ve completed your work, you MUST submit your changes as a git patch.”