A written disaster-recovery plan for a homelab, and the audit that admits most restores were never tested

Purpose

Having snapshots is not the same as having a recovery plan. The homelab has a written disaster-recovery (DR) plan, and a daily job that checks the homelab against it.

The plan has four goals. For every service, know what it needs to come back. Know that fresh copies exist (the RPO, or recovery point objective). Know that a restore works and how long it takes (the RTO, or recovery time objective). Know the order to bring things back in. The agents report and test. They never restore over production.

The plan was written on 2026-10-02 with a gap table. Most layers were unprotected that day: k3s cluster state, host configuration, secrets, restore tests, runbooks and the bring-up order. Production ZFS data and Proxmox guests were already covered.

Deployment

Tiers

Every service sits in a tier, and each tier has an RPO and an RTO.

Tier	Name	RPO	RTO	What is in it
0	Foundation	24 h	4 h	Network and DNS, the main Proxmox host and pools, both backup servers, Gitea, the k3s control plane, ingress, tunnel and VPN
1	Important	24 h	24 h	Home Assistant, Immich photos, Postgres, Frigate, personal data
2	Useful	7 days	3 days	Media, Jellyfin, the arr stack, Ollama and agents, Graylog, monitoring
3	Best effort	no target	no target	Tools, rebuilt from git

Bring-up order

Gitea comes before anything that builds from git, and ArgoCD syncs the apps from Gitea once k3s is up.

flowchart TD
    A["1. Power, router, switch, DNS"]
    B["2. Main Proxmox host and pools"]
    C["3. Proxmox Backup Server"]
    D["4. Gitea<br/>before anything that builds from git"]
    E["5. Docker hosts"]
    F["6. k3s, control plane first"]
    G["7. ArgoCD syncing apps from Gitea"]
    H["8. Hermes and the agents"]
    X["Daily audit"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    X -->|"RPO, offsite, last restore test, gaps"| A

The last step is also a check. The Hermes digest and the ZFS agent confirm the state of everything below them. See the ops digest post and the agents overview.

The plan follows the workloads

The plan was rewritten as workloads moved. k3s, Graylog and Hermes moved to the second Proxmox host on 2026-10-25, and Gitea moved the same day (see the host migration post). The first host was cleaned of those guests on 2026-11-02. The old guests were archived into a protected namespace on PBS until 2026-12-31 as the rollback path before being destroyed.

Configuration

The plan adopts a short list of practices:

• Tiers with RPO and RTO targets.
• 3-2-1-1-0: three copies, two media, one offsite, one immutable or air-gapped, and zero errors proven by restore tests. The immutable copy was still open when the plan was written.
• Rebuild from git.
• Back up the state git can’t hold.
• Encrypt what leaves the house.
• Test restores on a schedule and time them.

The daily audit

The audit uses no LLM and is read-only. It compares each backup stream’s newest copy with its tier’s RPO. It also reports offsite status, whether the offsite copy is encrypted, restore-test age (limit 90 days), rebuild-from-git, and known gaps. The tiers, streams and drill dates live in one hand-edited JSON file.

Drills

Planned agent DR3 runs drills monthly, rotating through targets. It restores only into isolated scratch targets and records the measured time per drill.

Drills done so far:

• 2026-10-18: the TrueNAS VM restored from PBS into a scratch VM in 100 s, and booted to a guest-agent answer 55 s later. The measured RTO is about 3 minutes plus pool import.
• 2026-10-18: the TrueNAS config export was decrypted and integrity-checked in 228 ms.
• Earlier: k3s state, host config and k3s volume restores were each tested from the offsite copy. Checksums matched, and row counts matched the live cluster.

Storage

What is backed up, and where:

• ZFS data. Production datasets replicate nightly to a local backup pool. A second machine pulls them offsite through a restricted forced-command user, so the offsite box can only read named datasets. Replication is verified by comparing snapshot GUIDs on both sides. See the offsite pull user post and the encrypted receive post.
• Guests. Proxmox guests back up nightly to Proxmox Backup Server. A second PBS on the other Proxmox host holds the migrated guests (Gitea, the k3s nodes, Graylog, Hermes), and its datastore is pulled offsite daily.
• State git can’t hold. Host configuration (Proxmox config, host /etc, local tools, units) and the TrueNAS middleware config are exported nightly into an encrypted bundle that is replicated offsite. The k3s etcd snapshot follows the same pattern.
• Recovery material. An offline kit exists. The audit tracks a verified date for each item and raises a high gap until someone checks it.

Access

The audit is a daily read-only report. The agents report and test, and they never restore over production.

Notes

What the audit found on 2026-11-02

On 2026-11-02 the audit listed 0 critical, 6 high, 20 medium, 6 low and 1 info findings. Readiness was 66 % for tier 0, 76 % for tier 1 and 45 % for tier 2. Two days earlier, on 2026-10-31, it was 60 %, 68 % and 27 %, with 11 high findings. These are dated numbers and they change daily.

Most of the medium findings read “restore not tested ever”. Every stream is fresh and offsite, but these have never been restored:

• guests such as Gitea, the k3s control plane and the Docker hosts
• both PBS copies
• the photo library and personal-data datasets
• the tier 2 streams

Other gaps on the list:

• a single k3s control plane
• no dead-man switch for Hermes itself
• router, tunnel and VPN settings not exported
• a workstation with no backup
• two load-balancer workloads not deployed by ArgoCD
• some tier 0 and tier 1 offsite copies not encrypted

A bug the audit work found

The nightly PBS verify wait had silently timed out every night since 2026-09-27, because the API token lacked a permission. The fix polls snapshot verification instead, and the digest now shows verify state.

What to take from it

Every stream is fresh and offsite, and most have never been restored. The audit reports that, and the open work is mostly drills.