Alerts that drown: acks with expiry dates and re-nagging

Purpose

The offsite backup NAS has a 3 x 12 TB raidz1 pool. On 2026-10-23 one member went REMOVED after repeated “SATA link down” messages. ZFS still reported “No known data errors”, so the pool kept working with no redundancy: one more failed disk would have lost it. Nobody noticed for about 3 days, and then only by accident, while setting up storage benchmarks on 2026-10-26.

Detection had not failed. The ZFS health agent’s facts were fresh, its rule (a non-ONLINE pool or a bad vdev state is a critical finding) fired, and the alert email transport worked. The alert drowned. This post covers the two small rules that fixed that: acknowledgements that expire, and a reminder every 7 days.

Deployment

The ZFS health agent is one of the daily agents described in the ops digest post. It produces findings with a severity, a daily report and a status page. Its settings live in zfs.json and its logic in zfs.py, both in the homelab-ops repo.

Two things made the first version easy to ignore:

• The status page had been red for days. Several critical findings were known and tracked but could not be fixed yet: a faulty NVMe path on the storage VM and a local replication task that kept failing. A page that is always red stops being read.
• Each critical finding alerted once and then went quiet. A fault that persists got one email and no follow-up.

On the day of the fix there were 9 active critical findings, most of them known and already tracked. Afterwards there were 2, both for the degraded offsite pool, and the rest were acknowledged.

Configuration

Acknowledgements with an expiry

Acknowledgements are a list in the agent’s config. Each entry has a glob key, an until date and a reason:

{"key": "zfs:<host>:pool:<pool>:*", "until": "2026-12-31", "reason": "known path fault, tracked"}

Keys match with fnmatch globs, so one entry covers a pool and all of its vdev and scrub findings. Validation rejects an entry that has no key or no until, so an acknowledgement can’t be open-ended. When until passes, the finding goes back to the active list by itself. A known issue that is still unfixed resurfaces instead of being hidden for good.

The config holds a few acknowledgements, each covering several findings through its glob. Their expiry dates are between 2026-11-30 and 2026-12-31 at the time of writing, and each reason names the tracking issue.

Re-nagging every 7 days

renag_days is 7. The report keeps an alerted_on map from finding key to the last day it alerted. A critical finding alerts when it is new, or when its last alert is at least renag_days old. An acknowledged finding is not in the critical list at all, so it never nags. An unacknowledged one never goes quiet.

flowchart TD
    A["Critical finding appears"]
    B{"Key matched by an<br/>unexpired acknowledgement?"}
    C["Acknowledged table<br/>no email"]
    D{"New, or last alerted<br/>7+ days ago?"}
    E["Send alert email<br/>record today in alerted_on"]
    F["Stays on Needs you<br/>no email"]
    A --> B
    B -->|"yes"| C
    B -->|"no"| D
    D -->|"yes"| E
    D -->|"no"| F
    C -->|"until passes"| B

Turning it on without a flood

A key that was already present on the previous run and has no alerted_on entry yet starts its clock on the day the feature is deployed. Switching it on therefore did not re-alert the whole standing backlog at once.

Storage

The feature adds the alerted_on map (finding key to last alert day). The pool it watches is the 3 x 12 TB raidz1 pool on the offsite NAS described above.

Access

The status page now has two parts. “Needs you” lists the active findings. “Acknowledged” is a table with the key, what it is, the until date and the reason, so an acknowledgement can be audited at a glance. Alerts go out by email.

Notes

• “We detect it” is not “someone sees it”. A permanently red page trains you to stop looking.
• Alert-once is wrong for faults that persist. An unfixed failure should keep reminding you, on a slow schedule.
• Acknowledgements need an expiry date so a known but unfixed issue comes back on its own.
• An acknowledged finding never nags, and an unacknowledged one never goes quiet.

Postscript: the cable

The hardware side took three commands: zpool status, dmesg | grep ata and smartctl -a. A hotplug re-seat failed, and a full power cycle brought the disk back. SMART showed 0 reallocated or pending sectors. Compare UDMA_CRC_Error_Count with the reallocated and pending counts: CRC errors point at the link (the SATA cable or connection), not at the disk surface. The pool resilvered back to ONLINE between 2026-10-26 and 2026-10-28.

That did not end it. The same disk dropped again about a week later, after a reboot (11-01). That resilver finished on 2026-11-01 with 0 errors. The SATA cable was replaced on 2026-11-04, and the pool is back ONLINE with all three disks and no longer degraded. The first scrub after the swap repaired checksum errors on that disk only, with 0 data errors. Those blocks had been written before the swap, while the bad cable was still in, and the link showed no new errors. A second scrub is running to confirm that the new cable fixed it. This time the re-nag keeps it visible. For the disk side of this kind of hunt, see the PCIe switch card post and attributing ZFS I/O delay.