We Ran ThruLine on Itself
AI coding agents wrote most of ThruLine’s code, and ThruLine recorded them doing it. This is what its own record caught, what it got wrong about itself, and one fix we recorded end to end.
This post is about our own product, so it follows our own rule: every number comes from ThruLine’s ledger, the repository’s history, the public v0.2.0 source, our internal engineering notes, the agent sessions’ own records, or a command you can watch run in the recordings. The notes say which, and §8 says how each figure was counted.
On 30 September at 05:13 UTC, ThruLine’s stop-time audit wrote a finding about ThruLine’s own repository. An agent had ended its turn with this:
“I read no standing instructions or RESUME.md, ran no workflow, and changed nothing.”agent’s final message · speaker_findings, seq 0
The audit filed that as a claim of work and searched the record for “an edit touching what the claim names.”1 It was wrong. The agent had said it did nothing, and asking it to prove an edit is a false alarm. For a tool like ours, a false alarm is the most expensive mistake there is.
It is also the only finding that audit has ever written about our repository. On 14 October we fixed it the way we now try to fix everything: start from the record, write the failing test first, and keep ThruLine watching the agent that does the work.
- ThruLine’s ledger on our build machine holds 588 sessions and 764,321 recorded events from building ThruLine. → §1
- The same rules caught our own mistakes: a false alarm, 20,020 false markers behind a locked database, a performance diagnosis that measured the wrong thing, and a review finding that didn’t reproduce. → §2, §3
- A decision you make once has to become a check. “No free trial” became a test and held. “Stripe, not Paddle” got one narrow test, and Paddle is still in a demo string and in our licence-issuing code. → §4
- On 14 October an agent fixed the false alarm in 2 minutes 35 seconds for $0.16. HQ held two of its test claims at Not proven until it ran those tests on their own. → §5
- We publish our failing rows: 31 of the 249 rows in v0.2.0’s sign-off check failed, and the release notes name each one. → §6
- What these stories leave open is listed in §7. → §7
§1Agents wrote most of ThruLine. ThruLine recorded them doing it.
ThruLine’s main development line has 1,941 commits between 9 August and 14 October 2026. 1,537 of them carry an AI co-author trailer: Claude Opus 5.5, Opus 5, Fable 5.1, Sonnet 5.5 and Fable 5, plus four from Codex.2 A trailer is what a commit says about itself, not a measurement, but it is a fair picture of who did the typing.
What we kept for ourselves were the rules, the reviews and the record. For large pieces of work, one agent session builds and a separate verifier session runs the result before it lands. Our rule is that work lands on a green gate: the tests, clippy with warnings as errors, rustfmt, and the UI’s tests and build. We haven’t always met it, and the public changelog says so where we didn’t. 90 commit subjects on the main line name the failing test before the fix, with words like “tests first” or “red at <commit>”. 63 commits record a break-and-restore: break the guard on purpose, watch the test fail, put it back.2
And ThruLine recorded all of it. On the machine ThruLine is built on, its ledger holds 123 Claude Code sessions (159,547 events) and 465 Codex sessions (604,774 events) from the repository’s folder, from 28 July to 7 October. From the Claude Code sessions alone it extracted 4,176 statements: 2,108 findings, 984 small intentions, 543 proposals, 401 completion markers and 140 claims of work done.3
§2Its first stop-time finding on our repository was a false alarm.
When an agent stops, ThruLine’s audit reads the agent’s last turn, pulls out the completion claims, and looks for proof of each in the session’s record. When it finds none for a claim it can check, it writes a finding. It is built to stay quiet when unsure. If a sentence can’t show that it reports the agent’s own work, the audit treats it as no claim at all, because silence costs less than a false accusation.
Part of that is a list of 34 hedges and negations. Any sentence containing one is dropped.4 Some entries end in a space, such as "nothing ", "zero " and "might ". The space was there to mark a whole word. It also meant that “nothing” at the end of a sentence, where a full stop follows it instead of a space, didn’t match.
/// Hedges and negations. Any occurrence kills the sentence outright. const HEDGES: [&str; 34] = [ "never", "zero ", "no longer", "nothing ", … ]; fn hedged(low: &str) -> bool { HEDGES.iter().any(|h| low.contains(h)) }
| Agent said | Audit |
|---|---|
| “…ran no workflow, and changed nothing.” | Finding |
| “…ran no workflow, and changed nothing at all.” | No finding |
“I edited RESUME.md.” | Finding |
The third row is right: nothing in an empty record shows that edit.
So the one sentence that says it did nothing, ending exactly that way, was treated as work. We reproduced it on the public v0.2.0 binary, checksum checked, with a fresh, empty ledger. §5 is the fix.
§3The same rules caught our own mistakes.
Three examples from our own history. In each, the fix started with measuring or reproducing, not with an opinion.
A locked database, and 20,020 false markers
On the night of 6 October, picking a project on ThruLine’s first-use screen sat on “Picking this project…” for minutes, then failed with SQLite’s own words: “database is locked.” The markers on disk showed why. Claude Code and Codex sessions are sealed into one shared store of saved copies. Each vendor’s reader checked that store against only its own accepted copies, so a Codex read treated every accepted Claude Code copy as an orphan. In four minutes it wrote 20,020 “unreferenced” markers, all of them false, while holding the lock the first-use screen was waiting for.5
v0.2.0 added a walk over both vendors’ copies, used when HQ reloads its reading. The comment beside it in the public source says the per-vendor ingest still marks the other vendor’s copies. And the first-use pick, the path that hung, still runs that per-vendor ingest in v0.2.0. That is open.
A performance diagnosis that measured the wrong thing
One of our slowest commands, packet --repo, took 159 ms against a 150 ms budget. An engineering session (A190) found a missing index and reported that about 85% of the cost was unexplained. The next session (A191) added that one index and measured again on the same 2,012,204-event ledger. packet --repo fell from 159 to 38 ms and now --all from 1,018 to 39 ms. The index alone accounted for 76% and 96% of their time.6
CREATE INDEX IF NOT EXISTS idx_nodes_project ON nodes(project_id), cleared both performance reds. The orange tick on each row is its budget, which applies to the 95th percentile: 168 and 1,074 ms before, 40 and 41 ms after.
A190 had timed the query in the sqlite3 shell, with a page cache the previous run had warmed. In A191’s own words: “The micro-benchmark was not measuring the same work as the surface it stood in for.” The next entry, A192, found the same problem twice more, including our instructions file quoting a twelve-minute cold export for three days after it stopped being true.
The fix was not only a faster query. Every performance figure in ThruLine now carries the basis it was measured on: corpus size, a digest of the engine source, the build profile and more. thruline perf checks that basis on the machine it runs on, and when the basis has moved it prints “stale basis — re-measure” in place of the number. A number printed next to a warning gets quoted anyway.
A review that dropped its own headline
On 13 October we pointed a reverse-engineering tool at v0.2.0’s shipped UI bundle and ran two red-team passes over its source. The first pass flagged a Markdown-injection hole in our pull-request comment renderer. Before writing it down, we ran 10 crafted payloads through cmark-gfm, the parser GitHub uses. None broke out, and the finding was dropped.7 Reproduce before you accuse is the rule ThruLine applies to agents. It applies to our reviews too.
§4A decision you make once has to become a check.
Two pricing decisions, two outcomes.
On 6 October we decided ThruLine has no free trial. The next day a test landed that fails the build if any shipped surface offers one: the licence, both READMEs, the Homebrew formula, the book, the UI files embedded in the binary, and the strings and doc comments the CLI and licence code ship, which is where --help text comes from.8 In v0.2.0 that search finds nothing.
On 26 September we chose Stripe over Paddle for billing. That decision got one narrow test: the Team screen’s live view must never say Paddle. Nothing checked the rest of the code. v0.2.0 shipped with “one Paddle invoice” in the same screen’s demo copy, Team-View/logic.js lines 271 and 280, and in the UI bundled into the binary.9 The public source has more: our licence-issuing crate’s README still says the founder “has not chosen between Lemon Squeezy and Paddle Billing”, its config lists those two providers, and a test asserts that Stripe is not one. That crate is only used in tests and is not in the binary. None of it is harmful, and all of it is exactly the drift ThruLine exists to catch: a decision we replaced, still in the code.
| Decision | Became a check? | What shipped in v0.2.0 |
|---|---|---|
| No free trial · 6 Oct | Yes, 7 Oct · no_trial_offer_guard.rs | No trial offered on any shipped surface |
| Stripe, not Paddle · 26 Sep | Partly · one test on the Team screen’s live view | “One Paddle invoice” in Team demo copy, logic.js:271, 280; Paddle and Lemon Squeezy in crates/licence-issue |
| The agreed HQ design | No | v0.1.0 screens drifted from it |
The third row is how v0.2 got its Decisions feature. Our v0.1.0 screens did not match the design we had agreed on, and nothing in our record said so, because nothing had turned that decision into something a machine could check. The v0.2.0 changelog puts the feature in one line: “a thing you said once becomes a check ThruLine runs on every version.”10
One more row belongs in that table. On our own build machine, ThruLine’s decisions table is empty. We built the feature and have not yet accepted a single decision on our own record. That is next, starting with Stripe.
§5End to end: the fix, and the claim HQ wouldn’t accept.
We took the 30 September finding and ran the loop we ask our users to run. All of this happened on 14 October, in a cloud workspace, on a clean clone of the public repository.11 The times are as HQ shows them, in IST (UTC+5:30).
- 21:24ThruLine’s hooks installed for the repository with
thruline init. - 21:27A Claude Sonnet 5.5 session starts with a written brief: the finding, the reproduction, and the rule to write the failing test first.
- 21:28The agent finds the trailing-space entries, writes a test that fails on the current code, runs it, sees it fail, and commits it.966ca3b · test result: FAILED. 0 passed; 1 failed
- 21:30It changes
hedged()so a whole-word entry also ends at punctuation or at the end of the text, commits the fix, and reports back.4463338 · 2 min 35 s · 10 turns · $0.16 - 21:30The stop-time audit runs on the report and writes nothing.
- 21:37HQ, reading the session: two of the agent’s test claims are not proven.
Not proven · version wt-7093e72 - 21:39Asked to run the extract tests on their own, the agent does: 22 passed, 0 failed.step 16 · 39 s · $0.20
- 21:40HQ reads the new step and moves both claims.
Proven · version wt-7093e72
The agent’s report made five statements. HQ held two of them at Not proven: “The new test and every other extract test pass” and “Extract tests: … 22 passed, 0 failed.” Its reason: “A later test run failed: step 7, 1 failed, 22 passed, on wt-9394b57.”
HQ was reading the record correctly, and the agent was telling the truth. In step 7 the agent had run the extract tests on its fix (22 passed). Then, in the same shell command, it set its work aside, checked out main, and ran one unrelated test that fails there too, to show the failure wasn’t its doing. One command, two versions of the code, one tally: 1 failed, 22 passed. A record of a command can’t be split into the halves the agent meant, so ThruLine didn’t guess. Not proven means the record doesn’t show it. It never means the claim is false.
Then HQ got something wrong. Its “Tell the agent” button copies a prompt asking the agent to re-run the step’s command, and the command it copied was that same compound one. Re-running it would have produced the same mixed tally. So we wrote the instruction ourselves: run the extract tests on their own, as one command, on the current code. The agent did. Step 16: 22 passed, 0 failed, on the same version as now. HQ moved both claims to Proven, on version wt-7093e72.
The other statements stayed where they should. Inside its report, the agent had quoted a sentence from its own test: “I edited RESUME.md.” HQ showed it as Recorded, the agent’s words with no verdict, not as an edit the agent claimed. “Failing test first…” and “Full library suite: 481 passed and 1 failed” are also Recorded: the agent’s words, with no verdict until someone says what counts as done for this change.
Then the patched build. The 30 September sentence no longer produces a finding. “I edited RESUME.md.”, with no edit in the record, still does (Figure 1, part 4).
And while recording Figure 1, we hit one more. Run thruline ingest against a ledger outside the default folder, on a machine that has never had a ~/.thruline, and it stops with “i/o error at …/.thruline/mirrors: No such file or directory.” Unless you pass --mirror-root, its folder of saved copies goes under your home folder, even with THRULINE_HOME set, which our README says moves that folder.12 It fails loudly, which is the safe way to be wrong, but it is wrong (Figure 1, part 5).
§6We publish our failing rows.
Before v0.2.0 we ran a census: an automated check that opens a test copy of ThruLine, presses every control the design draws, and compares what happens with what the design says. It has 249 rows. At each row’s newest measurement, 210 passed, 31 failed, 4 couldn’t be measured, 1 stopped on the check’s own setup, and 3 were left out of 0.2.0 on purpose. The seven core screens account for 136 of those rows: 133 passed, none failed, 3 left out on purpose. The release notes list every row that didn’t pass, by id, with one plain line each.13
The notes also correct themselves. An earlier draft said 211 passed and 30 failed; one row was measured again on newer code and failed. And one row caught a real bug hours before the release was signed off: just after midnight, three screens could misjudge whether the bar for “done” was set before or after the work, because they compared clock times. They now compare recorded moments. The upgrade from 0.1.0 was rehearsed on a clone of a 2.76-million-event record: 36 seconds, with every existing table’s rows unchanged apart from the upgrade’s own schema stamp.
§7What is still open.
- OpenThe stop-time fix from §5 is a two-commit patch in our working clone, not yet merged or released.
- OpenHQ’s “Tell the agent” can repeat a compound command that cannot settle the claim it is about.
- OpenA shell command that changes the checked-out code partway through is recorded as one run on one version.
- Open
thruline ingestputs its saved copies under the home folder by default, whateverTHRULINE_HOMEsays, and fails on a fresh machine with a ledger outside the default folder. - OpenPer-vendor ingest, which the first-use pick still uses, marks the other vendor’s saved copies as unreferenced. The markers carry no authority, but they are false.
- Open“One Paddle invoice” is in the Team demo copy, the licence-issuing crate still names Paddle and Lemon Squeezy, and no check covers the Stripe decision beyond one screen.
- Open31 census rows failed in v0.2.0, all outside the seven core screens. The release notes list them.
- OpenOur own decisions table is empty.
- OpenThe newest session transcript in our own ledger started on 7 October. The v0.2 work since then has not been read in.
- OpenThe 13 October review found four more rough edges in v0.2.0 that aren’t about the record. They are tracked for the next release, and the release notes list v0.2.0’s other known issues.
§8How we counted, and what this doesn’t show.
- Commits.
git logon our private main development line atc4d6d497(14 October). AI co-author means aCo-Authored-Bytrailer naming Claude or Codex. Test first means a commit subject containing “test first”, “tests first” or “red at”, in any case. Break-and-restore means the hyphenated phrase appears in the commit message. Builder and verifier roles are defined in the repository’s.claude/agents/. - Ledger. Read-only queries against the ledger on our build machine, opened in SQLite’s immutable mode so nothing was written. Sessions and events are counted for the repository’s folder, Claude Code and Codex. Statements are the ledger’s claim rows for the Claude Code sessions. The live hook table was being written while we read, and is not counted.
- Figure 1, first frame. A trimmed export of the finding row. The values are the ledger’s. It adds a
tablefield, showsproject_keyasprojectwith the project’s name, and leaves out the row id, session id, transcript path and line, the evidence count, the sensor bounds and two hashes. - §3. From our engineering log (A190 to A192), our incident notes from the night of 6 October, and our 13 October review. The two-vendor walk and its comment are in the public source,
crates/core/src/ingest.rs; the first-use pick is incrates/cli/src/uiserve/first_use.rs. - §5. Agent durations, turns and costs are the sessions’ own result records. The “Tell the agent” text was read from the clipboard by our recording script.
- §4 and §6. The public v0.2.0 source, changelog and release notes.
- What §5 doesn’t show. One session, with a brief we wrote, on a scratch clone. It shows how the loop behaves. It doesn’t show how often agents overclaim; that needs many sessions and labels set by people who don’t work on ThruLine.
A tool that tells you to ask for the record should be able to show its own. This is ours, including the parts we haven’t fixed.
Check your last agent sessionNotesSources
- ThruLine ledger on our build machine,
speaker_findings, seq 0, recorded 2026-09-30T05:13:23Z, chain hash0721a4ea…3aae44. Read-only. - Private development history, main line at
c4d6d497: 1,941 commits from 2026-08-09; 1,537 with an AI co-author trailer (Opus 5.5: 598, Opus 5: 379, Fable 5.1: 267, Sonnet 5.5: 169, Fable 5: 120, Codex: 4); 90 test-first subjects; 63 break-and-restore commits. Roles:.claude/agents/coder.mdandverifier.md. - Ledger, read-only:
sessionsandeventsfor the repository folder (Claude Code: 123 sessions, 159,547 events; Codex: 465 sessions, 604,774 events; first session 2026-07-28, newest 2026-10-07);claimsby kind for the Claude Code sessions. - Public source at v0.2.0,
crates/core/src/extract.rs:HEDGES(34 entries) andhedged(). github.com/thrulinehq/thruline - Internal incident notes from the night of 6 October 2026 (“First-use pick: database is locked”). Public source at v0.2.0:
crates/core/src/ingest.rs,LoadMirror::openand its comment on the per-vendor walks;crates/cli/src/uiserve/first_use.rs, where the pick callsingestoringest_codex. - Engineering log entries A190, A191 and A192 (private). A191 basis: 90 projects, 2,012,204 events, release build, 20 runs. Budgets: 150 ms for
packet --repo, 100 ms fornow --all.thruline perfships in v0.2.0. - Internal review of v0.2.0, 13 October 2026. Payloads rendered through
cmarkgfm; 0 of 10 broke out of the inert cell. - Public source at v0.2.0,
crates/cli/tests/no_trial_offer_guard.rs, with its full list of scanned surfaces. - Public source at v0.2.0,
ui/src/hq/screens/Team-View/logic.js, lines 271 and 280; the string is also in the bundledui/distassets. The live view’s test isteam-view-live.test.js. Licence-issuing crate:crates/licence-issue/README.md,products.toml, and the test insrc/lib.rs; it is a dev-dependency of the CLI. CHANGELOG.mdat v0.2.0, entry “2026-10-08 — Decisions engine”.- Agent transcripts, 14 October 2026, model
claude-sonnet-5-5, with ThruLine’s hooks installed bythruline init; branchfix/speaker-negated-work(966ca3b,4463338) on a clone of the public repository at825b365. HQ isthruline uifrom the v0.2.0 release. Commit ids are from our working clone and are not pushed. The “Tell the agent” text was read from the clipboard and saved. - v0.2.0 README, “What it records, where, and what leaves your machine”: “
~/.thruline/(setTHRULINE_HOMEto move it) … sealed copies of ingested logs inmirrors/”. Release binary: v0.2.0 release,thruline-cli-x86_64-unknown-linux-musl.tar.gz, SHA-2563b94eeaa…9674e. docs/release/v0.2.0-NOTES.md, “Census known issues”, andCHANGELOG.mdat v0.2.0 (midnight fix9a33c88a,0b4d0b97; upgrade rehearsal).