Counting Agent Runs and Model Spend: Local GPU vs Cloud

Purpose

Once a dozen agents and timers run on one box, “what ran, did it need a model, which one, and what did it cost” has no obvious answer. The question came up as four parts: how many times an agent ran, whether it needed a model, which model, and whether that model was the local GPU or the cloud. The answer should be in Grafana and in a daily Gitea report.

A small collector answers it from records that already exist, so no agent had to change. This post covers how it works and what it found. The agents themselves are described in the Hermes agents overview and the docs pipeline post. The eval harness post is related work: it grades what the agents write, this one counts what they cost.

Deployment

The collector runs on the Hermes host as two systemd timers. One collects every 15 minutes. The other runs at 06:40 and posts the daily report to Gitea. Both run as the hermes user, with journal read access limited to these two units. Both timers are active.

It reads four sources, all read-only:

• Job runs: the start and end lines the run wrapper writes to the systemd journal. The journal covers every cron job, systemd timer and webhook re-index.
• Model sessions and tokens: each Hermes profile’s session store.
• Local GPU calls: Ollama’s request log lines in Graylog (see the Graylog post).
• Cost: a price table in the collector’s config.

flowchart LR
    A["systemd journal<br/>job runs"]
    B["Hermes session stores<br/>models and tokens"]
    C["Ollama request log<br/>in Graylog"]
    P["Price table"]
    D["Collector<br/>every 15 min"]
    E["Postgres<br/>agent_usage"]
    F["Grafana dashboard<br/>Agents and models"]
    G["Daily Gitea report<br/>06:40"]
    A --> D
    B --> D
    C --> D
    P --> D
    D --> E
    E --> F
    E --> G

Configuration

Each source needed a rule to turn its records into counts.

Job runs: a kb-reindex run that exits non-zero because the repo is on the deny list counts as skipped, not failed.

Model sessions: the session store records the model, provider, API base URL, API calls and tokens. A local base URL means the GPU (Ollama). The Anthropic API host means cloud. The session title names the docs card, which is how cost can be split per card.

Cost: Hermes recorded $0 for Sonnet 5.5 because its own price list does not know the model. The collector computes cost as tokens times a price table in its config. The table has three entries: Haiku, Sonnet and Opus 5.5. The cache-write price is an assumption, 1.25 times the input price. Every cloud figure below is an estimate.

Local GPU: the Ollama log has no model name, so the config maps each client to the model it is configured to use. Local calls cost $0 in the report.

Storage

The data lives in a database called agent_usage in the k3s Postgres (see the Postgres post). It has four tables: job runs, LLM sessions, daily GPU requests and collector runs. The collector creates them itself.

Graylog keeps about 8 days of the Ollama log. Postgres keeps the history.

Access

The dashboard is called Agents & models and has 17 panels. It uses a Postgres datasource in Grafana (the Grafana post). The dashboard ships as a ConfigMap from the Grafana repo’s k3s branch and ArgoCD syncs it. All 17 panel queries were run against the data and each returned rows. ArgoCD needed a refresh annotation to pick up the merge.

The 06:40 timer posts the daily report to Gitea. It lists cloud spend for the last 24 hours, 7 days and the month, failed job runs, the busiest jobs, local GPU use, and cloud cost per docs card.

Notes

All figures are as of 2026-11-02.

What the first backfill showed (the journal only goes back to 10-20): 1,039 job runs (885 ok, 152 skipped, 2 failed), 104 model sessions and 1,689 GPU requests.

The surprise was that all 10 Hermes cron jobs are plain scripts with no model. The only cloud model use is the docs worker on Sonnet 5.5. Totals in the usage database: 73 Sonnet 5.5 sessions since 2026-10-22, about $15.68 estimated, including the day’s 11 briefs and 3 drafts.

Average cost per Sonnet session, by kind of card:

Card	Average	Runs
Brief	$0.35	27
Draft	$0.20	20
Publish	$0.07	17
Fact check	$0.29	3 (range $0.21 to $0.44)

The 56 local qwen sessions ($0, 2026-09-28 to 10-26) were the docs pipeline plus early experiments. 34 were docs cards on the local model before the switch to Sonnet. 22 were early Hermes runs in the default profile, before Hermes became a doc writer with no chat. Scheduled jobs never used a model.

The daily report for the last 7 days shows docs-queue-sync as the busiest job at 439 runs and kb-reindex at 225 (132 ok, 93 skipped), with 0 failed runs.

The local GPU handled 1,496 requests over 7 complete days. Most are embeddings (nomic-embed-text:v1.5) from the knowledge-base ingest (1,305 embed calls, 214 s of GPU time) and the knowledge-base MCP servers. The rest is a laptop client calling generate (162 requests, 16 errors). That client first showed up as an unknown app. It turned out to be editor autocomplete on a small coder model. Nothing is installed on the laptop to count it, so the numbers come from the server-side log.