ntfy on k3s: send-only push alerts from Hermes to a phone

Purpose

ntfy is a small self-hosted push notification server. In the lab it carries alerts from the Hermes agents to a phone. The email transport of the ops digest wasn’t finished, and the agents needed a push channel that stays send-only. ntfy became the second channel of Hermes’s notify tool, and Gatus watches it from outside as a dead-man check.

Send-only is the point of the design. Hermes only publishes to ntfy and nothing subscribes on its side, so a message sent to the topic cannot start an agent run. The no-chat rule holds.

Deployment

ntfy runs as one Deployment in the default namespace of the k3s cluster, with the image binwiederhier/ntfy:v2.32.0. It starts with serve and reads its config file from a ConfigMap. ArgoCD manages it as the app ntfy from the k3s branch of its repo, and the app is Synced and Healthy.

The Deployment strategy is Recreate. The data volume is ReadWriteOnce and holds SQLite databases, so two pods must never run at once.

A ClusterIP Service fronts the pod, and a Traefik IngressRoute sits in front of the Service. There are two entry points: a LAN address that Hermes publishes to, and a public one through the k3s Cloudflare tunnel for the phone app.

A Reloader annotation restarts the pod when the auth Secret or the config changes. Renovate opens image bump pull requests, and they are never automerged on the k3s branch.

flowchart LR
  H["Hermes notify tool"] -->|"token, write"| N["ntfy on k3s<br/>(Traefik)"]
  G["Gatus"] -->|"token, write"| N
  N --> T["alerts topic"]
  N --> CF["Cloudflare tunnel"]
  CF -->|"read"| P["Phone app"]
  N -->|"poll hint only"| S["ntfy.sh"]
  S --> I["iPhone"]
  N -->|"stdout logs"| V["Vector"]
  V --> GL["Graylog"]
  A["ArgoCD"] --> N

Configuration

Access is deny-all by default and signup is off. There are three users on one alerts topic:

• The owner reads, in the phone app and on the web.
• Hermes writes, with a token used only by its notify tool.
• Gatus writes, with its own token. It is the external dead-man: it alerts when Hermes is down or when the daily digest heartbeat is missing.

Users, access rules and tokens are provisioned at every start from a SealedSecret. To change a credential, reseal it with kubeseal. Reloader then restarts the pod.

For iOS the config sets an upstream base URL:

upstream-base-url: https://ntfy.sh

With it, iPhone pushes arrive instantly. The public ntfy.sh service only receives a poll request carrying a hash of the topic, never the message. The app then fetches the message from this server.

Storage

The data is a 1Gi PVC on the local-zfs storage class. It holds the message cache, which keeps 72 hours of messages, and the user database. The PVC has Prune=false in ArgoCD, so it survives deleting the app.

Access

The owner reaches ntfy from the phone app through the Cloudflare tunnel, or on the web. Hermes publishes over the LAN address. Every client authenticates with its own account or token, and nothing is readable anonymously.

Logs are JSON on stdout. The Vector DaemonSet ships them to Graylog (see the Graylog post). A Gatus check on the health endpoint monitors the service, in the style of the Gatus push checks.

Notes

To roll back, delete the ArgoCD app, remove the tunnel hostname and its DNS record, and switch ntfy off in Hermes’s notify config. The PVC stays until someone deletes it by hand. Mail alerts keep working throughout.

The work is tracked in homelab-ops #221.