Grading the Agents: An Eval Harness and a Tool Policy for Hermes

Purpose

The Hermes agents run unattended: the daily ops digest, the health board and the doc-writer cards. Until recently nothing measured how well a run went, and the only limits on tools were reactive (block after repeated failures) or hand-built read-only wrappers. Two small additions change that: an observe-only scorer for finished runs, and a declarative tool policy checked before a tool runs. The idea came from looking at the Turnstone platform. The decision was to borrow the ideas and not adopt the platform.

Both pieces follow one rule: measure before and after a change, so it is judged by data. For background see the agents overview and the docs pipeline. The code lives in the hermes-agent-plans repo under hermes-eval/ and hermes-tool-policy/.

Deployment

flowchart LR
    A["Agent run"] --> B["agent.log"]
    A --> C["Observer hooks"]
    B --> D["Shadow collector"]
    D --> E["eval.jsonl"]
    C --> F["metrics.jsonl"]
    C --> G["decisions.jsonl"]

Everything runs on the docs profile. Doc writing uses Claude Sonnet 5.5; the agent and tool work uses a local qwen model.

The scorer is a stdlib-only Python script. It reads one finished run’s slice of agent.log and prints a JSON record. It never runs inside the agent and cannot block, slow or fail a run. A parse miss produces an “incomplete” record, not an exception. A systemd service and timer run a shadow collector every 15 minutes. It scores each finished run and appends a line to eval.jsonl, and it skips runs it has already scored. It only reads the agent log and writes only inside its own directory.

Configuration

Each run is scored as a vector: API calls, tool calls, peak context tokens, output tokens, latency, wall time, models used, repeats, tool errors, whether it was hard-stopped, and the end reason. Two flags sit on top:

• failed: the outcome was bad, the run was hard-stopped or interrupted, or it had errors.
• degraded: repeated calls, context over budget, or wall time near the run cap.

A third field flags regressions against the agent type’s trailing median once there is history. The first backfill scored 34 real runs: 10 failed and 14 degraded. Two doc-brief runs that hit the 45-minute cap came out as interrupted, failed and degraded, which matched a manual audit.

The tool policy is a manifest per profile. Each tool is tagged read or write and allow or deny, using the same classification Hermes uses for its own tool sets. Unknown tools count as write-capable. A hook checks the manifest before a tool runs, and a deny comes back to the agent as a normal blocked-tool result. The manifest has three stages:

mode: describe   # log only
mode: observe    # log every decision
mode: enforce    # deny can take effect

The docs profile now runs in enforce mode with every rule set to allow. That changes nothing in behaviour, and it proves the hook fires. It was validated on a real published post. The manifest’s own comments say its tool list is still incomplete, so no deny rule goes in until the decision log has covered many real cards.

Two further observer hooks record each API request: tool count and estimated input size before the call, then duration, token usage and finish reason after it. Observers cannot fail closed, so they cannot affect a run.

Storage

The three JSONL files are plain local files on the agent host: per-run scores, per-call decisions and per-API-call metrics. The collector is idempotent and the policy hooks are removable by deleting the hook block and their directory.

Access

There is no interface yet. The files are read on the host. A small eval panel on the health board is the planned next step.

Notes

The metrics paid off on the first experiment. Re-enabling deferred tool loading was tried with the observers recording both arms. The numbers showed a regression, so it was rejected.

Next steps are comparing the local qwen model with Sonnet on the same kinds of runs, and the first real deny rule, for example a write tool in a read-only profile.