Local qwen vs Claude Sonnet for a doc-writing agent: what 44 scored runs showed

Purpose

Grading the Agents built an observe-only scorer for the Hermes agents and named one comparison as its next step: the same doc-writing workload on a local model and on a cloud model. This post is that comparison, and a look at how far it can be trusted.

The doc-writer agent started on a local 27B qwen model running on the lab’s own GPU. Draft cards often hit the 45-minute card limit, and some never finished. On 2026-10-24 drafting moved to Claude Sonnet 5.5. Because the scorer had been recording every run, the switch could be judged with data instead of impressions. For the surrounding setup, see the docs pipeline post and the agents overview.

Deployment

The scorer is observe-only. It reads each finished run’s slice of the agent log, and a systemd timer appends one JSON line per run to eval.jsonl every 15 minutes, skipping runs it already scored.

flowchart LR
    A["Agent run"] --> B["Agent log"]
    B --> C["Shadow collector<br/>15-minute timer"]
    C --> D["eval.jsonl"]
    D --> E["compare.py"]
    E --> F["Table: qwen bucket<br/>and Sonnet bucket"]
    G["Card id<br/>not yet recorded"] -.->|"needed for a paired A/B"| F

Each record holds API calls, tool calls, peak context tokens, output tokens, latency, wall time, the models used, repeats, tool errors and the end reason. Two flags summarize it:

• failed: a bad outcome, a hard stop, an interrupt or errors.
• degraded: repeats, context over budget, or wall time near the card cap.

Draft and publish cards moved to Sonnet on 2026-10-24, and brief cards on 2026-10-26. The profile’s default model is still the local qwen, and it applies to anything without a per-card model.

Configuration

compare.py takes eval.jsonl and buckets runs by model family. A run that used both families counts as escalated to Sonnet. It prints counts, failed and degraded rates, and medians for wall time, API latency, API calls, output tokens and peak context.

The recorded comparison from 2026-10-26 covered 44 runs:

	qwen	Sonnet 5.5
Runs	33	11
Failed	30%	0%
Degraded	42%	not recorded in this table
Median wall time	1719 s	27 s
Median API latency	1588 s	20 s
Median API calls	15	6

That is about 65 times faster by wall clock. At drafting time (51 runs, 33 qwen and 18 Sonnet) Sonnet still had 0 failures and the speed-up read about 48 times. The table keeps moving as runs are added: Sonnet’s median call count has since gone from 6 to 8, so the post doesn’t claim it uses half the calls.

Peak context was nearly the same for both families, so context size doesn’t separate them. Time per call and the number of calls do.

Storage

The cost of a post comes from token counts. The first Sonnet card, on 2026-10-24, looked like this:

Card	API calls	Model time	Estimated cost
Draft	8	44 s	about $0.19
Publish	5	14 s	about $0.03

That is about $0.22 per post, and about $0.33 with one review-fix round. The figures are estimates from token counts, not from the provider’s usage page. The prices used were $2 and $10 per million input and output tokens, with cache reads at $0.20 and cache writes at $2.50. From the second call on, 72 to 99% of input was a cache hit.

The hand check for one run is cache reads × $0.20/M + cache writes × $2.50/M + output × $10/M.

The same card on qwen the day before: the brief took 22 minutes, then the draft timed out at 45 minutes after 27 calls.

Access

A $20 monthly spend cap was set on 2026-10-29. At $0.22 to $0.33 per post it covers roughly 60 to 90 posts. A card stuck in a loop is the bigger cost risk, and the 45-minute card limit bounds it.

One side benefit: docs cards no longer evict the voice assistant’s model from the shared GPU (the Intel B60 described in the voice assistant post).

Notes

• The split is a before/after, not an A/B. It tracks the 2026-10-24 switch, and the qwen runs span 2026-09-30 to 2026-10-26. Anything else that changed that week is mixed in with the model. A clean A/B needs a card id in each record so the same task can be paired. compare.py says the same in its footer.
• Degraded stayed similar. 42% for qwen against 28 to 36% for Sonnet, depending on the day. The guard rails fire on the task as well as on the model.
• Speed doesn’t replace fact checks. Sonnet’s first drafts still had weak spots, such as a wrong root cause and a vague count, that a review round fixed.
• Token counts don’t compare across families. Claude’s tokenizer counted 25,105 tokens on a first call where qwen counted 22,700 for the same text.
• Tool-error counts are left out. Counting started on 2026-10-26 and only one of the 33 qwen records has the field, so there is nothing to compare.

The takeaway for policy is to let a looping local run escalate to the cloud model. The scorer already records an escalation field. Next is the paired A/B by card id.