The share that emptied itself: receiving ZFS into an encrypted pool with -x encryption

Purpose

I moved datasets from the Proxmox host to a TrueNAS VM with zfs send and zfs recv. The first one, a small SMB share, copied cleanly and passed every check. About an hour and a half later TrueNAS rebooted and the share was empty. This post covers the cause, the receive flag that prevents it, the flag that has to stay off, and a checklist to run before cutover.

Deployment

The target is the main pool of the TrueNAS VM. It is encrypted at its root with aes-256-gcm. The pool is its own encryption root, and the other datasets on it (ISOs, config export, VM data) inherit that encryption.

The pilot dataset was one SMB share with no app attached, 583 files in 132 directories and 33 snapshots at seed time. The seed was a full zfs send -p plus an incremental (-I). It was relayed through another host, because the Proxmox host had no SSH trust to TrueNAS yet.

Cutover was verified before the share went live: the file listing (path, size, owner, mode, mtime) and a sha256 over all contents were identical on both sides.

Configuration

What went wrong

A -p stream carries the source’s properties, and the source was unencrypted, so the stream carried encryption=off. The dataset arrived unencrypted inside an encrypted pool. On every pool import, TrueNAS runs its unencrypted-datasets handler, which force-unmounts unencrypted children of an encrypted root. After the reboot the middleware logged:

Successfully umounted any unencrypted datasets under '<pool>'

The share pointed at an empty directory. The data was never touched. zfs mount <pool>/<ds> brought it back at once, but only until the next import.

flowchart TD
  A["Source dataset<br/>(unencrypted)"]
  B["zfs send -p"]
  C["recv without -x encryption"]
  D["Dataset lands with encryption=off"]
  E["TrueNAS import"]
  F["Child force-unmounted"]
  G["Empty share (BAD)"]
  H["recv -x encryption"]
  I["Dataset inherits the pool key"]
  J["Import mounts it"]
  K["Share intact (GOOD)"]
  A --> B
  B --> C
  C --> D
  D --> E
  E --> F
  F --> G
  B --> H
  H --> I
  I --> J
  J --> K

The fix: -x encryption, no -e

-x encryption excludes the property from the stream, so the new dataset inherits encryption from its parent. The seed looks like this:

zfs send -L -c -p <src>@<oldest> | ssh <truenas> zfs recv -s -u -x encryption <pool>/<ds>
zfs send -L -c -p -I @<oldest> @<newest> | ssh <truenas> zfs recv -s -u -x encryption <pool>/<ds>

-s makes the receive resumable. -u keeps the dataset unmounted until cutover. Dropping -p from the send also works.

Do not add -e (embedded blocks) to the send. An encrypted receive refuses it with “incompatible embedded data stream feature”.

A raw send (-w) keeps the source’s encryption. The source here was unencrypted, so -w would also have produced an unencrypted dataset under the encrypted root. It only helps when the source is already encrypted.

Repairing the dataset in place

The pilot was already live, so I repaired it on TrueNAS itself, with the same send and receive into a new dataset named <ds>.enc:

1. Receive <ds> into <ds>.enc with -x encryption: full send of the oldest snapshot, then -I to the newest.
2. Disable the share briefly and send one more incremental snapshot.
3. Make the old dataset read-only, unmount it, and rename it <ds>.unencrypted-old with canmount=noauto.
4. Rename <ds>.enc to <ds>.
5. Re-apply zfs allow for the pull user and re-enable the share.

The new dataset was aes-256-gcm with the pool as its encryption root and keystatus available. ACL type, xattr, compression and aclinherit settings were kept. It had 36 snapshots with identical task user properties, and old and new matched on both a listing hash and a sha256 over all contents.

Storage

Receiving into the encrypted pool keeps each snapshot’s identity. Both backup chains, a local pull to another host and an offsite pull, continued with incrementals after the swap and needed no full resend. The next scheduled runs of both succeeded and matched TrueNAS. The offsite side is described in the offsite pull post. It uses the same pattern: a non-raw stream into a new child of an encrypted root is written encrypted with the parent’s key.

The middleware saw the new dataset as encrypted and not locked. After a later VM restart it mounted on its own, so the fix holds across an import.

Every later wave used the corrected method. The largest, a 157 G dataset, was seeded from the Proxmox host to TrueNAS in about ten minutes (06:13 to 06:23, up to 390 MB/s, 35 snapshots), also aes-256-gcm with the pool as encryption root.

Access

Encrypted writes were slow because of the VM’s CPU type. The guest ran x86-64-v2-AES, which exposes aes and sse4_2 but not PCLMULQDQ, AVX/AVX2, AVX-512 or VAES, so ZFS AES-GCM used its slow generic path. A local encrypted write went from 110 MB/s to 866 MB/s with cpu: host, and read from 72 MB/s to 2.6 GB/s (partly ARC). The details are in the CPU post.

Notes

Checklist for receiving into an encrypted pool:

1. Check the target: zfs get encryption,encryptionroot <pool>.
2. Receive with -x encryption (or send without -p), never with -e. Add -s and -u.
3. Before cutover, run zfs get encryptionroot,keystatus <pool>/<ds>. The encryption root must be the pool and keystatus available.
4. After any swap, compare snapshot guids on both sides of each backup chain.
5. If the VM is slow, check that it sees the host CPU flags (cpu: host) before blaming ZFS.

A fresh migration looks fine for hours, because TrueNAS only acts on the problem at the next pool import.