A Second Model Reads the Draft Before the PR Exists: Fact-Checking Hermes Posts

Purpose

The offsite ZFS post shipped with “12 TB” in its Purpose section and “about 13.5 TB” in its Notes. All five automatic checks passed: post, sanitize, markdownlint, mermaid and gitleaks, run on the exact pushed bytes. None of them compares a claim with a source, so none could see the mismatch. The fix was a one-line change to the sentence, “which forced a resend of about 13.5 TB”, and it landed the same evening.

A check on the open pull request would have been too late. Brad merged Hermes docs PRs (15 of them) a median 4 minutes after they opened, and merge is publish. So the fact check sits between the draft and the PR. This post describes how that works, the numbers from the first live runs, and where it still falls short. It builds on the docs pipeline and the agents described in the Hermes overview.

Deployment

The shape is draft, check, publish. Each step is its own card on the docs queue.

flowchart LR
    A["Docs request"] --> B["Draft card"]
    B --> C["Check card<br/>fresh Sonnet session"]
    C -->|"pass for this exact file"| D["Publish card"]
    D --> E["Pull request"]
    C -->|"findings: one re-draft"| B
    C -->|"second failure"| F["Blocked, Brad decides"]

The check is a fresh Claude Sonnet 5.5 session running a separate skill, fact-checker. It does not see the writer’s reasoning, so it is not anchored on it. It compares every claim in the post with the brief, Brad’s answers, the cited primary records and the writer’s facts file.

Each claim gets one result: ok, mismatch, unsupported, or leave-out violation. The check also reports contradictions inside the post itself, which is the class the 12 TB bug belonged to.

Configuration

A small deterministic tool, post-check, is the gate. It records the result of the check and posts one comment with the claim table on the idea issue. The publish tool then refuses to open a new PR that closes a docs request unless the check passed for the exact normalized bytes of the file. Edit the file after the check and the pass no longer applies. The PR body gets a line like “Fact check: pass, N claims”.

Brad made these decisions on 2026-11-02:

• A failed check blocks publishing.
• unsupported claims are listed but do not block. Mismatches and leave-out violations do.
• After a failure the draft gets one automatic re-draft with the findings. If that fails too, the card blocks for Brad.
• Drift-fix PRs are not checked, and neither are review-fix updates to an existing PR.

The queue turns findings into one fix card plus a recheck, with publish placed behind the recheck.

Storage

The result is recorded by the gate tool and shown as the claim-table comment on the idea issue. Where the gate keeps its records stays out of this post.

Access

The publish step opens the PR, and it needs the pass described above for the exact file.

Notes

Acceptance test: the original 12 TB draft, checked against its brief and comments. It came back with 22 claims ok and 1 contradiction (Purpose against Notes and the brief), in 32 seconds on Sonnet 5.5. The gate correctly refused to record it, because the request it belonged to was already closed. Test counts at deploy: gate tool 24/24, publish tool 50/50, brief tool 30/30, queue graph 6/6.

First live results, from three docs requests that ran draft, check, publish:

• 87 claims, 0 blocking, 4 without a source.
• Of those 4, three held up when compared with the records. One was a real error that the checker could only list: a SQL index name, chunks_vec where the code said chunks_vec_idx, found later in a PR review.

So the check does not replace a reviewer for facts that have no source to compare against.

Cost: the plan estimated $0.05 to $0.10 per check. The three live checks cost $0.21 to $0.44, about $0.29 on average on Sonnet 5.5, from the agent-usage database.