Config Compliance: Timezones, Logging, Provenance and a Repo Inventory

Purpose

Four daily, report-only checks run against every Docker and k3s workload in the lab. Each answers a basic question: is the timezone set correctly, do logs actually reach Graylog, was this deployed from Git or by hand, and does every repo follow the branch rules? The pipeline uses a Hermes agent pattern where deterministic scripts gather facts, pure functions decide what is a finding, and rolling issues in a private ops repo update only when something changes.

The four checks live in compliance.py, a stdlib-only script with about 50 unit tests. Each generates a Markdown report that report-issue posts to its own issue. If an issue empties out, it closes itself instead of sitting half-empty.

Architecture

flowchart LR
    dh["Docker hosts\n:05:12 facts push"] --> cr
    k3s["k3s API\n:read-only SA"] --> cr
    argo["ArgoCD apps"] --> cr
    gray["Graylog\n:7d last-seen"] --> cr
    gitea["Gitea repos\n:inventory"] --> cr
    cr["compliance-run\n:05:20 daily"] --> c1
    cr --> c2
    cr --> c3
    cr --> c4
    c1["TZ compliance"] --> ri
    c2["Logging coverage"] --> ri
    c3["Deploy provenance"] --> ri
    c4["Repo inventory"] --> ri
    ri[("Rolling issues\nprivate ops repo")]
    rb[("repo-steward: rsteward, write only\ncreates missing k3s branch")] -- "write" --> c4

Inputs arrive from five sources. Docker hosts push structured fact files via forced SSH commands (each host can only submit its own), kubectl queries return workloads and ConfigMaps reduced to their TZ key, ArgoCD reports applications, Graylog returns last-seen counts for the past seven days, and repo-steward inventories every Gitea repository. A failing input produces an .error file instead of data; the check still runs with what it has and reports the gap.

Reports are stable without dates or volumes, so report-issue only updates when something changed since the last run. The four rolling issues drop per-finding noise entirely: when everything is clean they close themselves.

Provenance classes

Deployment provenance is a prerequisite for automatic fixes. You can only auto-fix what is tracked in Git. Each workload falls into one class:

flowchart TD
    start(["Workload"]) --> kind{"Docker or k3s?"}
    kind -->|Docker| label{"Compose label set?"}
    label -->|"yes, clean git clone"| d1["Git-clean compose"]
    label -->|"yes, local changes"| d2["Git with host changes"]
    label -->|"no label, in git"| d3["Compose not in git tracking"]
    label -->|"plain docker run"| d4["docker run (manual)"]
    kind -->|k3s| a{"ArgoCD app?"}
    a -->|"yes"| k1["ArgoCD GitOps"]
    a -->|"no"| h{"Helm release?"}
    h -->|"yes, values in git"| k2["Helm (values tracked)"]
    h -->|"yes, values local only"| k3["Helm (values not tracked)"]
    h -->|"no"| u{"kubectl apply?"}
    u -->|"yes, last-applied in git"| k4["kubectl from git"]
    u -->|"yes, files on host only"| k5["kubectl from files"]
    u -->|"no"| m[("k3s add-on or<br/>operator managed")]

Docker workloads are classified by compose directory state: clean git clone, clone with local changes, compose not tracked in git, a plain docker run, or a Portainer stack. k3s workloads sort through ArGoCD, Helm (with or without values in Git), kubectl apply (from Git or host-only files), or platform-managed add-ons and operators.

Permissions and roles

Each part runs under its own identity with only the rights it needs, enforced outside the LLM so a compromised model cannot escalate:

• compliance-run runs as hermes with no write privileges. The kubectl context (hermes-ro) is a read-only service account: no Secrets, no exec. Docker facts are in setgid-hermes directories readable but not writable by the hermes user.
• report-issue runs through a sudo gate under the gmcp Unix user, who owns a write token scoped only to four rolling issues. The tool validates report names against an allowlist, checks file ownership, blocks symlinks, and runs gitleaks before posting.
• repo-steward runs as its own rsteward user with a Gitea token restricted to Docker deployment repos. The token can create branches on those repos and read the rest, never delete, rename, or overwrite. It only creates a missing k3s branch from main.
• ops-facts-receive runs as opsfacts. SSH authorized_keys pins each host to its own fact file via forced commands: the pushing host names what it sends, but cannot choose a different filename.

Hermes can only call write tools through sudo rules that allow exactly one binary each. The determinism gate (compliance.py as a pure function with unit tests) means the LLM never makes decisions about findings.

How it runs

Four agents run daily in no-agent mode: scripts execute, output is delivered, no LLM turn involved. On Hermes, hermes cron schedules these jobs and delivers the results into their respective channels. The morning order is:

1. 05:10 Docker hosts push facts via forced SSH (ops-facts-receive). Each host connects to its own name on the allowlist; stdin must be valid JSON under 2 MB with a host field matching the name.
2. 05:00 docs-queue sync picks up any new documentation issues.
3. 05:20 compliance-run gathers k8s, Docker, Graylog, and repo inputs, runs compliance.py through all four checks, then posts to rolling issues via report-issue. A --dry-run flag validates without posting.
4. 05:30 ops digest consumes the same facts for its own reports.

Check 1 (timezone) flags containers where TZ is wrong, missing, or unverified because a Secret holds the value. After fixes land through the R2-3 fixer agent, no container remains with a wrong timezone; the remaining work is always finding ones with no TZ at all. Check 2 verifies that each workload ships to Graylog (GELF log driver, Vector sidecar, or file shipping like Traefik) and that Graylog has actually received logs in the past seven days. Quiet services are allowed low counts; the rule is presence, not frequency.

How it relates to the other agents

See the Hermes agent overview for the full series. The compliance pipeline shares its input layer with the ops digest — both use the same pushed Docker facts, kubectl output, and Graylog queries, just processed through different scripts. The provenance check identifies which findings can be auto-fixed because their manifests are in Git, and which require manual changes to compose files or host-level configurations. Every agent follows the same pattern: deterministic collectors make facts, rules decide on findings, and a root-owned gate tool validates before anything leaves Hermes.

Notes

Platform workloads (CoreDNS, kube-proxy, Longhorn system pods, ArgoCD itself) are exempted via glob patterns in compliance.json so they do not clutter the app tables. When an exempt group grows large enough it shows in a collapsible section below the main report instead of disappearing entirely. The log shipper sidecars (Vector) appear in the workload list but are excluded from logging coverage checks: the question is whether the application ships, not whether the shipper itself reaches Graylog.

The repo bot creates only one kind of change: a k3s branch missing from a deployment repo that already has the docker topic and a main branch. It never deletes or overwrites. Adding docker topics requires a human decision, which is why the bot does not touch them.

---

See also: Hermes agent overview for how Hermes enforces these constraints.