Flashing a one-way BIOS on a Proxmox host that depends on its own passthrough VM

Purpose

This is the checklist I used to update the BIOS on my main Proxmox host from 2.03 to 3.04, and the step I missed the first time. The host is awkward for firmware work: it runs its storage server as a VM, and it also mounts storage that the VM serves back to it. The flash went fine. The first poweroff hung because of that circular dependency.

Deployment

The board is an ASRock Rack SIENAD8-2L2T with an AMD EPYC 8024P (8 cores / 16 threads) and 64 GiB of RAM. BMC firmware 03.01.00 was already current.

The storage server is a TrueNAS VM (see the TrueNAS VM post). Its 3 NVMe controllers are passed through to it. The host mounts NFS and NVMe-over-TCP storage that this VM serves.

The update was BIOS 2.03 (2025-02-12) to 3.04. The download is dated 2026-03-30, and dmidecode reports a release date of 01/23/2026. It brings AGESA GenoaPI 1.0.0.H, PSP 00.29.01.A4 and a new “Primary Display” option.

Firmware Anti-Rollback enforcement was Enabled, so going back to 2.03 is very likely impossible. That is why the preparation below is longer than for a normal update.

Configuration

Before: snapshot, backups, kernel pin

1. Check the night’s backups and save fresh host config.
2. Snapshot the BIOS settings over Redfish with a read-only BMC account: GET /redfish/v1/Systems/Self/Bios, plus the attribute registry for the menu paths. I saved the raw JSON, a readable TSV and a list of non-default settings. The snapshot had 1,014 settings and only 9 differed from defaults: boot order, boot beep, anti-rollback state, one CBS value, serial-over-LAN settings and a BMC host interface setting.
3. Record the pool error counters. The host had an open NVMe error investigation, so I wanted a before value.
4. Pin the running kernel so the reboot changes only the firmware:

proxmox-boot-tool kernel pin <current>
proxmox-boot-tool refresh

Shutdown order

Stop the guests in this order: cluster workers, control plane, other containers, and the storage VM last. Confirm qm status <vmid> says stopped.

Before stopping the storage VM, there is one more step, and it is the one I missed:

• disable the NFS storage and the NVMe/TCP storage in Proxmox
• umount the NFS path
• nvme disconnect the subsystem

The storage plugin for the NVMe/TCP storage rejects pvesm set, so that storage can’t be disabled that way. The disable line has to be added to the storage config directly, and removed again afterwards.

flowchart TD
    A["Pre-flight: backups and host config saved<br/>Redfish settings snapshot<br/>pin kernel, record pool errors"]
    B["Stop guests in order<br/>workers, control plane, other containers"]
    C["Disable NFS and NVMe/TCP storages<br/>umount, nvme disconnect"]
    D["Stop storage VM"]
    E["Poweroff and flash"]
    N["Hangs if the host still mounts<br/>storage from the stopped VM"]
    F["Boot and diff the settings"]
    G["Check IOMMU groups and vfio"]
    H["Start storage VM"]
    I["Re-enable storages, unpin kernel"]
    A --> B
    B --> C
    C --> D
    D --> E
    E -.-> N
    E --> F
    F --> G
    G --> H
    H --> I

Storage

On 2026-10-21 the guests were stopped between 17:45 and 17:48. I had skipped the disable and unmount step, so the poweroff at 17:48 hung. With the storage VM already down, the hard NFS mount and the NVMe/TCP disconnect both blocked. After more than 8 minutes I powered the host off from the BMC, flashed the BIOS, and the host was up at 18:17. All guests were up by about 18:21.

The same logic now runs automatically as a Proxmox VM hookscript on the storage VM. In the pre-stop phase it unmounts the NFS paths and disables the storages. It disconnects NVMe/TCP only when no running guest has a volume on that storage, and otherwise just warns. It records what it disabled. In the post-start phase it re-enables what it recorded.

Access

The settings snapshot used Redfish with a read-only BMC account. The BMC address and account details stay out of this post.

Notes

After: diff and passthrough checks

The diff between 2.03 and 3.04 showed 7 changed, 1 gone and 11 new settings. All 9 custom settings were kept. Re-Size BAR and SR-IOV were turned on by the new defaults, and the boot entries changed to “Linux Boot Manager”. Having the snapshot made “what did the flash reset?” a diff instead of a guess.

After boot, before starting the storage VM:

dmidecode -t bios
ls /sys/kernel/iommu_groups | wc -l
journalctl -k -b | grep AMD-Vi
lspci -k
lspci -vv

Those check the BIOS version, the IOMMU group count, AMD-Vi, vfio-pci ownership, and link speed and ASPM. The result was 74 IOMMU groups, AMD-Vi active and vfio-pci owning the 3 NVMe controllers again. Then I started the storage VM and checked that the pool imported.

What went differently

• The kernel pin did not hold. Before the flash the host ran 7.0.14-20-pve, the pinned kernel. The newer 7.0.14-25-pve was picked at the boot menu, so the BIOS and the kernel changed together. I kept the newer kernel, and no kernel is pinned now. Check the running kernel after boot instead of assuming.
• A host that mounts storage from its own VM can hang at poweroff. Disable those storages first.
• Two failed units after boot looked alarming but were already failing before the flash. Compare with the previous boot before blaming the change.