Moving a 1.7 TB dataset without history: a two-snapshot seed plan, a scrub that said no, and the 160 GB move that shipped

Purpose

The camera recordings dataset had to move from a QLC SSD pool on the Proxmox host to an encrypted NVMe pool in a TrueNAS VM. It held about 1.69 T of live data plus 73.6 G in 13 snapshots, unencrypted, recordsize 128K, with a compression ratio of about 1.05x because it is video. The recorder is the one in the Frigate post.

Two replicas already existed, a local backup pool and an offsite NAS. Each held only the snapshots from its own replication task, and both kept one week. Neither should take a full 1.7 TB resend.

This post covers the plan, why it did not survive, and the smaller move that shipped on 10-31.

Deployment

The plan, decided on 10-22: carry no snapshot history. Seed exactly two snapshots, then let both replica chains continue incrementally from the new home.

• B is the day’s newest local-backup snapshot, sent as a full stream.
• O is the day’s offsite snapshot, newer than B, sent as an incremental with -i. That is about an hour of video.

Before the real work I measured the write rate. It came to about 20 G per day (about 0.25 MB/s), far below the “tens of MB/s” I had first assumed. The seed itself was the heavy write, not the recorder.

flowchart LR
  SRC["Recordings dataset<br/>QLC pool, Proxmox host"]
  BKP["Local backup pool<br/>holds B"]
  OFF["Offsite NAS<br/>holds O"]
  DST["Encrypted NVMe pool<br/>TrueNAS VM"]
  GATE{"Scrub clean?"}
  HOLD["No: hold, 10-22"]
  CUT["Yes: cutover, 10-31"]
  AGE["Seed ages out<br/>after 1 week"]
  FRESH["Fresh seed, 160 G"]
  SRC -- "seed B (full), then O (-i)" --> DST
  BKP -- "B" --> DST
  OFF -- "O" --> DST
  DST --> GATE
  GATE --> HOLD
  GATE --> CUT
  HOLD --> AGE
  AGE --> FRESH
  FRESH --> CUT

Configuration

The seed used a resumable receive that stays unmounted and read-only until cutover:

zfs send -L -c -p <src>@B | ssh <truenas> zfs recv -s -u -x encryption -o readonly=on <pool>/<ds>
zfs send -L -c -p -i @B @O | ssh <truenas> zfs recv -s -u <pool>/<ds>

-x encryption and no -e, because the target pool is encrypted. Before sending, each snapshot was checked on its replica for the same guid as on the source (zfs get -H -o value guid <ds>@<snap>).

The cutover was planned as: stop the app, snapshot @migrate-<date>, send it with -i @O (not -I, which would carry intermediate snapshots), compare guids, rename the old dataset .pre-truenas read-only as a rollback, re-point the container mount, restart.

The gate before cutover was zero NVMe errors during the seed, then a scrub with 0 repaired. Any error meant stop.

Storage

The target pool had 6.81 T free. The seed finished on 10-22 at 19:41: 1.68 TB received, both snapshots guid-equal with the source, encrypted under the target pool, 0 NVMe errors. The app kept recording the whole time.

The go/no-go scrub then triggered a fatal PCIe error on the NVMe carrier card and a host kernel panic. The gate said no-go. The app was untouched and kept recording on the old pool. The fault and its fix are in the PCIe switch card post, and the receive details are in the encrypted pool post.

By 10-31 the seed could not be used. The replicas’ one-week retention had pruned both seed snapshots on the source, so no common snapshot remained and an incremental cutover was impossible. The trick only works inside the retention window, and a long hold means re-seeding.

While looking at the data again I found that the recorder’s retention had silently not worked since 06-29. A cleanup thread crashed on a permission error in 15 empty directories left owned by host root, mode 700, inside an unprivileged container. The disk held 1.8 T instead of the configured window.

After fixing the permissions and cutting retention from 21 to 7 days, the dataset dropped from 1.81 T referenced to 160 G. The old snapshots still held 1.76 T until replica retention pruned them.

Access

The cutover ran on 10-31 at 16:01 with a fresh seed of the 160 G snapshot. The full send ran at about 46 MB/s, limited by the QLC source pool. After the carrier swap, a 200 GiB soak write had run at 1.3 GB/s with 0 errors, and the 10-30 scrub found 0 errors. The final scrub repaired 0 B in 5 min 34 s. A random sample of 200 files from the cutover snapshot had the same sha256 summary on both sides.

The steps: app stopped, final -i send guid-checked, old dataset renamed .pre-truenas read-only, container mount re-pointed. The container was down about 20 s.

One change from the plan: Brad dropped both old replicas. The B/O chain was never used. Both replicas were re-created from the new home with a full 160 G send, and the first nightly runs on 11-01 were incremental from them, guid-equal on both sides. The offsite side is described in the offsite pull post.

The recorder now depends on the TrueNAS VM and NFS. If TrueNAS is down, recording stops until it is back, and a mount check alerts. Brad accepted that.

Notes

• The app container did not start after the host container restarted. The script stopped the container first, and restart: unless-stopped leaves a stopped container off. The recording gap was about 1 minute, and the script now starts the container explicitly.
• Two replicas with different snapshot names can both stay incremental if the seed carries one common snapshot for each. That only holds within the retention window.
• Measure first. The real write rate changed the gates, and looking at what was on disk shrank the move from about 1.8 T to 160 G.
• A seed that runs while the old copy is live doubles as a soak test, but the scrub after it is the real test of the new hardware path.
• The TrueNAS dataset is now 283 GB used, consistent with the 7-day window.