Exposing AES-NI to a TrueNAS VM: cpu: host and encrypted ZFS throughput

Purpose

My TrueNAS lives in a VM on Proxmox, with its NVMe drives passed straight through so ZFS owns the hardware directly. The pool is encrypted with aes-256-gcm. For a while the encrypted write throughput felt low for what the drives can do, and the fix turned out to be a single line in the VM’s config: the CPU type.

The change

The VM had its CPU type set to x86-64-v2 — a portable, conservative baseline that boots on almost anything. I changed one setting to host, which hands the guest the physical CPU’s real instruction set instead of a generic model. Nothing else changed at the same time — no extra RAM, no NIC swap, no recordsize tweak.

Why it matters

x86-64-v2 is an older microarchitecture level: it has SSE4.2 and friends, but no AES-NI and no AVX2. ZFS encrypts with aes-256-gcm, so under x86-64-v2 the guest was doing all of that crypto in software. Setting the CPU type to host passes through the host CPU’s full feature set — AES-NI and AVX2 included — so ZFS gets hardware crypto instead.

The general lesson: hardware crypto and vector instructions only reach the guest if the hypervisor’s CPU model actually exposes them. A “safe” generic CPU type can quietly leave a big chunk of the host’s capability on the table.

flowchart LR
  A["NFS / SMB client"]
  B["TrueNAS VM"]
  C["ZFS aes-256-gcm dataset"]
  D["passthrough NVMe pool"]
  E["cpu: host (AES-NI + AVX2)"]
  A --> B
  B --> C
  C --> D
  E --> B

What changed

The headline result is the encrypted write: it went from about 110 MB/s to about 866 MB/s — roughly 8× — purely from exposing AES-NI to the guest. That is the clean, CPU-attributable win, and it matches the mechanism: software AES is the bottleneck, hardware AES removes it.

End-to-end NFS reads also rose (from about 42 MB/s to about 1.2 GB/s), but I’d read that number carefully rather than credit it all to the CPU. 1.2 GB/s sits right at my 10 GbE line rate, so the “after” read is bounded by the network, not by crypto — once the write path stopped being the bottleneck, reads simply ran up against the link. The honest framing is: the write speedup is the CPU story; the read number mostly reflects the network ceiling.

Notes

• The trade-off of cpu: host is portability: a guest pinned to the host CPU can only live-migrate to an identical (or compatible) CPU. For a single-host storage VM that never migrates, that cost is zero.
• The numbers here are from my own before/after measurements on this box; treat them as “what I saw,” not a benchmark suite. The write result is the one I’d stand behind as a CPU effect.
• If you want to separate CPU from network on the read side, measure a cold read (larger than ARC, so it can’t be served from cache) on the encrypted dataset, and check the raw link with a network test first — that tells you which ceiling you’re actually hitting.