Gatus can't see that mount, so the host tells it

Purpose

Gatus polls most of the lab from inside the cluster: ICMP and HTTP status checks, plus a DNS check per service hostname. It has no view of the mount table on the Proxmox host, so it can’t tell whether a TrueNAS NFS share is still mounted there.

For those mounts the direction is flipped. The host runs the check and pushes the result to Gatus, and Gatus treats “no push in time” as a failure. The same pattern works for anything a poller can’t reach.

Deployment

Gatus runs as one replica of twinproduction/gatus:v5.38.0 in the default namespace, configured from a ConfigMap. On the host, a systemd timer fires every 2 minutes and runs a small script that checks each TrueNAS NFS mount and pushes one result per mount.

flowchart LR
  T["TrueNAS VM"] -. "NFS over the network" .-> M["NFS mounts on the Proxmox host"]
  A["systemd timer on the Proxmox host (every 2 min)"] --> B["check each TrueNAS NFS mount"]
  M --> B
  B -- "push result with token" --> C["Gatus external endpoint (group Mounts)"]
  C --> D["10 min heartbeat window"]
  D --> E["push arrived: healthy"]
  D --> F["no push: failed"]
  F --> G["alert / ops digest"]

Gatus in the cluster has no line to the mount table. That is the reason for the push.

Configuration

Gatus has an external-endpoints: feature for this. Each endpoint declares a token and a heartbeat interval, and Gatus marks it failed if nothing arrives within that interval:

external-endpoints:
  - name: <host>-truenas-Pictures
    group: Mounts
    token: "${PUSH_TOKEN}"
    heartbeat:
      interval: 10m

The timer pushes every 2 minutes against a 10 minute window, so about five pushes land in each window and one lost push doesn’t turn the endpoint red.

Gatus resolves the token from an environment variable fed by the cluster Secret, so it knows the value to expect. The host script reads its own copy from a local token file. The token never appears in the config or in this post.

An endpoint’s key is built from the group and the name, lowercased, with a few punctuation characters replaced by -. The name in the config therefore decides which URL a push has to target.

What the check tests

The script does two things per mount:

1. It confirms the mount comes from the expected TrueNAS export.
2. It runs stat -f on the mount under a 10 second limit (timeout -k 2 10), which only succeeds if the NFS server answers.

The second step is the point of the whole setup. A hung NFS mount stays in the mount table, so a check that only asks whether the share is mounted passes while df, pct list and pvestatd sit in uninterruptible sleep. The bounded statfs catches a mount that is present but not responding, and reports mounted but not responding (statfs timed out). The migration plan for moving storage to TrueNAS names a per-mount check like this as the mitigation for that failure.

Only mounts that have a guest list in the host’s mount list are pushed. A dataset nobody uses isn’t monitored, so it can’t go red for no reason.

Storage

Two mounts are covered today, both in the Mounts group: the Pictures share and the Pictures upload share. Both are TrueNAS NFS shares mounted on the Proxmox host. The mount list has a third entry, which has no guests and isn’t pushed.

Access

The host authenticates each push with the token described above. The results appear in Gatus under the Mounts group.

Notes

An external endpoint is only useful once something pushes to it. A third endpoint, for the Frigate storage mount, was declared in the Gatus config before that mount existed on TrueNAS. It never received a push and stayed red until the declaration was removed. It goes back in together with its line in the host’s mount list when Frigate’s storage moves to TrueNAS.

Declare the endpoint and the pusher in the same change.