Hermes Agents: How a Local AI Ops Crew Is Wired

This is the overview for a series on the agents that run automated operations inside this homelab. The first post covered the agent itself: its guardrails, identities, and how access works. This one covers the team: six mostly LLM-free agents following one pattern, sharing the same constraints, all reporting to the same place.

Purpose

Instead of one general-purpose agent doing everything, each task runs under a tight identity with only the permissions it needs. The scanner can read the cluster but write only its own results; the docs worker can open a pull request but never merge it; the compliance checks report findings, and the only change they make themselves is adding a missing k3s branch to a repo. The design keeps the language model away from anything that writes state: write access goes through gate tools that validate inputs, run linting and security scans, and produce artifacts (issues or pull requests) a human reviews.

Architecture

The system has three layers: facts, agents, and gates.

Facts. Hosts push their own state to JSON files on a daily schedule (timezone, log drivers, ports, mounts, networks). The Proxmox host and the Docker hosts push theirs at about 05:10 local time; if a file is stale, that is a finding. The k3s cluster doesn’t push anything: the agents read it through a read-only service account.

Agents. All but one run as Hermes cron jobs; the compliance fixer runs on demand. Each one reads facts, applies rules, and produces output:

Agent	Schedule	Output
Ops digest	daily 05:30	Rolling issue updated only when findings change
Config compliance	daily 05:20	Four rolling issues (timezone, logging, provenance, repo inventory)
Kubescape scan + triage	daily 06:00	Scan results + baseline comparison fed to ops digest
Docs queue sync	daily 05:00	Kanban board populated from briefs
Docs drift detector	weekly Sunday 03:00	Rolling issue listing stale facts in posts
Compliance fixer	on-demand	Pull requests for the human to review

Gates. Two tools handle publishing. report-issue updates a rolling Gitea issue in place: it reads the report, validates file ownership and size, runs gitleaks (plus sanitize-check for public repos), then creates or updates the issue. publish-post does the same for documentation posts, pushing to a fork and opening or updating a PR. Both are sudo-gated: each cron job’s identity can call exactly one wrapper binary.

flowchart LR
    subgraph Hosts[Infrastructure]
        PXE[Proxmox host]
        D1[Docker hosts]
        K[k3s cluster]
    end

    subgraph Collectors["Read-only collectors"]
        F["Facts files (~05:10 daily)"]
        KK["kubectl (read-only SA)"]
        GR["Graylog query"]
    end

    subgraph Agents[Scheduled agents]
        OD[Ops digest 05:30]
        CC[Config compliance 05:20]
        KS[Kubescape scan + triage 06:00]
        DD[Docs drift weekly Sun 03:00]
    end

    subgraph Docs[Docs pipeline]
        QS[Queue sync 05:00]
        BW[Brief writer]
        DW["Doc writer (LLM step)"]
    end

    subgraph Gates[Gate tools]
        RI[report-issue]
        PP[publish-post]
    end

    F --> OD
    F --> CC
    KK --> OD
    KK --> CC
    GR --> OD
    KS --> OD

    OD --> RI
    CC --> RI
    DD --> RI

    DW --> PP

    subgraph Issues["Rolling issues (private repo)"]
        OPS[Ops digest]
        CMP[Compliance]
        DRF[Drift report]
    end

    RI --> Issues

    PP --> PRs[("Pull requests (owner reviews + merges)")]

    classDef infra fill:#3a3a3a,stroke:#555,color:#ccc;
    classDef collecting fill:#2a3a4a,stroke:#446,color:#9cf;
    classDef ag fill:#2e3a2e,stroke:#464,color:#9f9;
    classDef dt fill:#3a2e3a,stroke:#656,color:#c9c;
    classDef gt fill:#3a3020,stroke:#762,color:#fb8;
    class Hosts infra;
    class Collectors collecting;
    class OD,CC,KS,DD ag;
    class QS,BW,DW dt;
    class RI,PP gt;

Permissions and roles

Every agent runs under its own identity. There is no master account:

• Kubernetes: a dedicated hermes-ro ServiceAccount with get, list, and watch on workloads, logs, and CRDs. No secrets access, no exec, no pod creation. Verified with kubectl auth can-i.
• Gitea (read): a restricted bot account with read-only scopes for repos, issues, PRs, and CI runs. The scanner’s write identity lives in a separate token under a different user.
• Publish: two system users hold the write tokens. Each has exactly one sudo rule allowing a single wrapper. The wrappers validate arguments (allowlisted report names and file paths), check file ownership, enforce no-symlink reads, run gitleaks, and push output as an issue body or PR commit.
• The compliance fixer runs locally on the workstation, not through Hermes at all. It generates a PR first; the host follows the merged repo using its own read-only deploy key, never the other way around.

flowchart LR
    subgraph LLM_side["LLM (Hermes session)"]
        MODEL["Qwen 3.6 27B (local, no cloud API)"]
        AGENT["Cron job or docs-queue card session"]
    end

    MODEL -->|prompt + tool call| AGENT

    subgraph Gates["Outside the LLM (sudo-gated tools)"]
        W1[report-issue wrapper]
        W2[publish-post wrapper]
    end

    AGENT -.->|"sudo (one binary each)"| W1
    AGENT -.->|"sudo (one binary each)"| W2

    subgraph Identities["Separate identities per task"]
        KSA[k8s read-only SA]
        GITEA_R[Gitea read bot]
        DOCS_T["Docs write token (held by a separate system user)"]
        SCAN_W["Kubescape write token (separate user)"]
    end

    W1 --> Identities
    W2 --> Identities

    classDef llms fill:#3e2a3a,stroke:#645,color:#f99;
    classDef gt fill:#3a3020,stroke:#762,color:#fb8;
    classDef id fill:#2a3a3a,stroke:#466,color:#acf;
    class MODEL,AGENT llms;
    class W1,W2 gt;
    class KSA,GITEA_R,DOCS_T,SCAN_W id;

How it runs

Hermes cron jobs handle scheduling. Facts gathering runs in “no-agent” mode: a plain script executes, outputs results, and that output is delivered, with no LLM turn needed for data collection. Jobs run daily with order enforced by offset:

1. ~05:00: docs-queue sync pulls new briefs into the kanban board
2. ~05:10: hosts push their facts files
3. 05:20: config compliance runs against fresh facts
4. 05:30: ops digest collects from all 12 sources, applies rule-based thresholds
5. 06:00: Kubescape scan with a read-only identity; triage step compares results to a 7-day rolling baseline

The LLM is only involved where prose matters: writing briefs from three-line ideas and drafting posts. The collectors, the rules and the reports, including the ops digest’s log triage, are plain scripts. Even for posts, the publishing step is a deterministic tool call with its own validation.

How it relates to the other agents

The data flows one direction: hosts push facts, collectors read, agents report. There are no circular dependencies between jobs except where that is intentional: the docs pipeline fixes drift findings, the fixer resolves compliance findings, and the ops digest watches all of them (did the kubescape job run on time, is any kanban card stale). The pattern is the same: find something, describe it, hand it off. No agent trusts another’s output at runtime because there’s no runtime consumer: every consumer is a human reading an issue or reviewing a PR.

Coming in this series

Each agent gets its own post covering the collectors it uses, the rules it applies, false-positive handling, and what happens when things go wrong:

• The ops digest, with log triage and the edge watch
• Kubescape scanning and baseline triage
• The docs pipeline, from a three-line idea to a reviewed PR
• The docs drift detector
• Config compliance: timezones, logging, provenance and the repo inventory
• The compliance fixer, from finding to reviewed PR

Notes

The trade-off is speed for safety. A finding discovered at 05:30 does not get fixed automatically: it sits in a rolling issue until I review it, decide on a fix, or acknowledge why it’s expected (acknowledgments are time-boxed; an expired one re-becomes a finding). In exchange, nothing runs with more permissions than it needs to complete its job, no secrets can escape the wrappers, and every write is a visible artifact that survives beyond a single session.