ZFS checksum errors that turned out to be a PCIe switch card

Purpose

A three-drive NVMe raidz1 pool kept logging write and checksum errors, and every error was on the same two drives. For about a week the drives took the blame. The fault was the Gen3 PCIe switch on the carrier card all three drives sat behind. This post covers how to tell a path fault from a drive fault, and why a fix that only hides the symptoms is dangerous.

Related posts: the TrueNAS VM post and the I/O delay post.

Deployment

The pool is 3 x 4 TB NVMe in raidz1: two Crucial P310 (DRAM-less) and one Lexar. The drives are passed through by PCI address to the storage VM.

The old path was a Sabrent EC-P3X4 carrier with an ASMedia Gen3 switch. The uplink trained at Gen3 x4 and each drive at Gen3 x2, about 1.75 GB/s each, all sharing about 3.5 GB/s. The root port feeding the card is capable of Gen5 x8 but trained at Gen3 x4 (measured on 2026-10-27, before the card came out).

The new path is an ASUS Hyper M.2 x16 Gen5 card with no switch, in an x16 slot set to x4x4x4x4 in the BIOS. Each drive sits on its own CPU root port, capable of 32 GT/s x4 and trained at 16 GT/s (Gen4) x4. The fourth bay is free.

flowchart LR
  subgraph OLD["Old path"]
    A["CPU root port<br/>uplink Gen3 x4"] --> B["Gen3 switch carrier card"]
    B --> C["Crucial P310 #1<br/>Gen3 x2"]
    B --> D["Crucial P310 #2<br/>Gen3 x2"]
    B --> E["Lexar<br/>Gen3 x2"]
  end
  subgraph NEW["New path"]
    F["CPU slot<br/>bifurcated x4x4x4x4"] --> G["Switchless card"]
    G --> H["Drive 1<br/>Gen4 x4"]
    G --> I["Drive 2<br/>Gen4 x4"]
    G --> J["Drive 3<br/>Gen4 x4"]
  end

Configuration

2026-10-20. Both P310s logged WRITE errors under load, with status sct 0x3 / sc 0x60 (Controller Pathing / Internal Path Error). sc 0x60 is generated by the controller and is not kept in the drive’s own error log. That points at something between the drive and the CPU, but at the time it read as a drive fault.

The first fix was ASPM L1 off on the host and NVMe APST off in the storage VM. Three clean scrubs followed. The pool looked healthy.

2026-10-21. The errors came back with ASPM and APST still off, during a 2 GB receive. That made 14 errors since 2026-10-17, all on the P310s, some on single 4 KiB writes at idle. The Lexar, the SMART counters and the AER counters on the used ports were clean. No firmware update exists for the P310s, so the working conclusion was still the drives.

2026-10-22 and 2026-10-26. The Proxmox host panicked with GHES: Fatal hardware error. The AER uncorrectable status was 0x00014000 on the first (Completion Timeout and Unexpected Completion) and 0x00114000 on the second (the same two plus Unsupported Request). Both were on the root port feeding the carrier card. The first outage lasted about 10 minutes and the second about 6 to 7, each with a remote power cycle.

A paused scrub that is resumed reads at full speed, which is the heaviest read load the pool can generate. It re-triggered the fault, so scrubs stayed paused until the hardware was fixed.

The commands that mattered:

# which vdev, which errors, which block
zpool status -v <pool>
zpool events -v

# in the storage VM: the status code the drive does not keep
dmesg | grep -i nvme

# on the host: compare what the port can do with what it trained at
lspci -vv -s <root-port>    # LnkCap vs LnkSta

LnkCap on the root port is the slot’s real electrical capability. Don’t trust the widths a switch card negotiates.

Storage

2026-10-27, the swap. The BIOS link-width menu is per slot. Only the x16 slots offer x4x4x4x4. The x8 slot offers only x8 or x4x4, so a switchless card there lights two bays. A three-drive switchless card needs a slot that bifurcates four ways.

Moving the card changes the PCI addresses, and a passthrough by address then fails with “no PCI device found” when the VM starts. Record the new addresses before starting the VM and re-map them:

qm set <vmid> --hostpci0 <new-address>,rombar=0

The pool imported with a latched DEGRADED flag. After zpool clear and a full zpool scrub, it repaired 0 B with 0 errors, and the drive that had been marked degraded showed 0/0/0. The last scrub, on 2026-10-31, was also clean, and all three drives show 0/0/0. Each drive went from Gen3 x2 to Gen4 x4.

Access

The drives are passed through by PCI address to the storage VM, so the VM sees them as plain NVMe devices.

Notes

• All the errors were on two of the three drives, which looked like a drive problem. Those two were the busiest members behind the faulty switch.
• ASPM and APST changes hid the fault for a day and made it look solved. Three clean scrubs did not mean the path was fixed.
• sc 0x60 was the early hint. When the controller reports a pathing error that the drive does not log, look at the link before the drive.
• Read LnkCap and LnkSta on the root port early. Here the port could do Gen5 x8 and ran at Gen3 x4.
• On a path with a fault, a resumed scrub is a trap. Leave it paused until the hardware is fixed.
• Passthrough by PCI address breaks on any card move.