Post

Your backup isn't done until you've restored it

Your backup isn't done until you've restored it

Purpose

There are two numbers in a recovery plan, and backups only answer one of them. RPO — recovery point objective — is how much data you would lose; a nightly backup that completes puts a ceiling on it. RTO — recovery time objective — is how long you would be down, and the only honest way to know it is to restore something and time it. This post is about a disaster-recovery drill whose restore was far too slow, and about the discipline that mattered more than the fix: measuring the restore, and refusing to trust the first explanation for a bad number.

The drill

The test was deliberately non-destructive. A roughly 142 GiB service container was restored from an encrypted Proxmox Backup Server (PBS) datastore — held on a small, low-power second backup box — onto scratch space on the main hypervisor. The restore ran with a temporary, read-only API token scoped to a single datastore namespace, created for the drill and revoked afterward; the datastore’s encryption key was pulled from a self-hosted secrets store into a temporary keyfile that was removed when the drill ended. The restore target was a throwaway guest that was never started (its network interface would have been removed before any boot) and was destroyed afterward — a “does it come back?” test, not a cutover.

flowchart LR
    A["Backup box<br/>encrypted PBS datastore<br/>(ZFS SSD mirror)"] -- "chunks over 1 GbE" --> B["Restore client<br/>on the main hypervisor"]
    B --> C["Checksum"]
    C --> D["Decrypt"]
    D --> E["Decompress"]
    E --> F["Write to scratch guest"]
    F -. "then request the next chunk" .-> A

The result was the whole point of running it: the restore read at about 55 MiB/s for its first 13 minutes, reaching about a fifth of the container, then collapsed to 6–8 MiB/s and stayed there. At that rate the full container projects to roughly four to five hours, so the drill was aborted on its time budget at 22%. A nightly backup I trusted was quietly attached to a half-day recovery time.

Reading the number

The interesting part was everything the slow number was not. It was not the network: the backup box sits on a 1 GbE link that stayed under 7% utilized. It was not disk seeks: the datastore is a ZFS mirror of two SSDs, not spinning disks. It was not deduplication: dedup is off on that datastore. And the serving box’s CPU, a low-power chip, stayed near idle throughout.

I list them all because my first written-down diagnosis — “random reads on a deduplicated HDD store” — was wrong on every count, and I had written it confidently. Two minutes of actually checking the hardware replaced it. The lesson travels well beyond backups: when a measurement surprises you, verify the premise before you reach for the cause.

Instrumenting both ends

When no single resource is pegged, you have to watch all of them at once. A useful restore has you sampling, on both the serving box and the restore client: per-core CPU — not the load average, which hides a single saturated thread behind three idle ones — plus link utilization read straight from the kernel’s byte counters, and zpool iostat -v for read IOPS, bandwidth and latency.

With nothing saturated, the signature points at the shape of the work rather than any one component: a serialized, per-chunk restore of an encrypted backup. Each chunk is a round-trip to fetch, a checksum, a decrypt, a decompress and a write — and if those happen largely one chunk at a time, a fast link and fast SSDs can’t help you, because nothing is ever busy for long. That is the hypothesis the next, instrumented drill is set up to confirm or kill.

Notes

  • RPO is not RTO. “The backup ran” is a statement about the first and says nothing about the second.
  • A cheap, low-power box is fine for holding backups. Whether it can restore fast enough is a separate question, and it is the one your recovery time depends on — so know which box that is.
  • Make restore credentials scoped and ephemeral, and keep the datastore encryption key somewhere other than the backup system itself.
  • Run restore drills on a schedule, time each phase, and set a hard abort time so a drill can never run into a backup window.
This post is licensed under CC BY 4.0 by the author.