A 25 GbE cable between two Proxmox hosts, and the NFS traps that came with it

Purpose

The two Proxmox hosts described in the Proxmox post now share a direct 25 GbE fibre cable with no switch in between. The main host holds the media pool and the TrueNAS VM (see the TrueNAS post). The backup host runs the k3s containers and mounts storage from the main host over NFS. If the cable dies, the mounts move to the 10 GbE LAN by themselves.

The cable was the easy part. Most of this post is about the four problems that appear when containers and pods hold copies of an NFS mount. For context, the same day the main host’s HBA was swapped for an LSI 9305-24i and the pools came back after a second round of recabling.

Deployment

Both cards are Mellanox ConnectX-4 Lx on the mlx5_core driver, negotiated at 25000 Mb/s. Each host has a Linux bridge vmbr1 on the 25 GbE port, with MTU 9000 on the port and the bridge, in a small separate subnet. The LAN bridge stays at MTU 1500. A jumbo ping (-M do -s 8972) passes both ways at about 0.15 ms (measured 2026-11-03).

The TrueNAS VM got a second virtio NIC on the main host’s vmbr1, hot-plugged without a restart, with mtu=1 so it inherits the bridge’s 9000. Inside TrueNAS it is ens19 at MTU 9000.

Before the switch, a test read over the link gave about 470 MB/s. That was limited by the main host’s HDD pool, not by the link. No throughput was measured after the switch.

flowchart LR
  B["Backup host<br/>vmbr1, watcher, k3s containers"]
  M["Main host<br/>vmbr1, media pool NFS export"]
  T["TrueNAS VM<br/>ens19, HandBrake shares"]
  B ==>|"25 GbE fibre, MTU 9000"| M
  M ==>|"vmbr1 bridge"| T
  B -.->|"10 GbE LAN, MTU 1500, /32 route when failed over"| M
  B -.->|"10 GbE LAN, MTU 1500, /32 route when failed over"| T

Configuration

A live NFS mount cannot change its server address, so the mounts always use the link addresses and failover is done with routes.

The watcher is a small bash daemon run by systemd (restart always). Every 5 s it pings each server’s link address, bound to vmbr1:

ping -c1 -W1 -I vmbr1 192.0.2.11

After 3 misses (about 15 s) with the LAN address answering, it adds a /32 route to that server via its LAN address on the LAN bridge, then kills the NFS TCP connection so the client reconnects at once on the new path:

ip route replace 192.0.2.11/32 via 198.51.100.11 dev vmbr0
ss -K dst 192.0.2.11 dport = :2049

After 6 good pings in a row (about 30 s) it deletes the route and kills the connection again. If both the fibre and the LAN are silent, it logs a warning and changes nothing.

This works because both servers accept their link address when it arrives on their LAN interface, and both export to the backup host’s LAN address as well as its link address.

A boot-time oneshot unit (--once, ordered before the three mount units) waits up to about 20 s for the fibre to train, then picks fibre or LAN per server before anything mounts.

In a live drill on 2026-11-03 the backup host’s 25 GbE port was taken down for 60 s while a reader made 783 reads from the media pool. The watcher moved both servers to the LAN 16 s after the link dropped, and moved them back 29 s after it returned.

sequenceDiagram
  participant W as Watcher
  participant R as Routing table
  participant N as NFS connection
  W->>W: 3 missed pings on the link (about 15 s)
  W->>R: add /32 route via the LAN
  W->>N: ss -K, client reconnects
  W->>W: 6 good pings on the link (about 30 s)
  W->>R: delete the route
  W->>N: ss -K, client reconnects

Storage

The backup host mounts three NFS shares over the link: the media pool from the main host and two HandBrake scratch shares from the TrueNAS VM. After its reboot the NFS client lists only the two link addresses.

The media pool mount is a systemd mount unit, NFSv4, hard, in place since the k3s move to the backup host. The HandBrake scratch shares are mounted on top of the media pool directory (see the Jellyfin and HandBrake post).

The k3s worker containers get the media pool through a Proxmox bind mount point. The TrueNAS export allows the host, not the containers’ VLAN addresses, so the NFS connection has to come from the host’s network namespace.

Because the mounts are hard, I/O pauses during a switch and then resumes, with nothing lost. In the drill the one read in flight when the link dropped waited 15.7 s and then completed, no read failed, and the switch back caused no visible pause.

Access

Only the backup host and its containers use the link. The k3s worker containers get the media pool through a Proxmox bind mount point, and pods reach it through hostPath volumes. The TrueNAS export allows the host, not the containers’ VLAN addresses, so every connection has to come from the host’s network namespace.

Notes

Four traps came with the setup.

1. NFSv4 trunking. Remounting the same server under a new address silently reused the old LAN session, because containers and pods still held copies of the old mount. Traffic only moved after a host reboot.
2. Lazy unmount under running LXC containers orphaned the containers’ copies of the mount, so they stopped receiving new submounts.
3. Proxmox bind mount points are non-recursive: submounts that exist before a container starts are missing inside it. After the backup host’s first reboot with these mounts, the HandBrake shares were missing inside the k3s worker containers. They were mounted by hand and the pods recreated.
4. Kubernetes hostPath volumes with default propagation keep the old view until the pod is recreated.

The fix for trap 3 is a Proxmox hookscript, post-start phase. It runs nsenter -m into the container’s mount namespace and mounts each share there. Only the mount namespace is entered, so the NFS connection comes from the host’s network namespace and reuses its session. The source is read from the host’s own mount, so it follows whichever path is active. It waits up to 120 s for the host mount and never fails the container start. It is attached to the three k3s worker containers. It is installed and is meant to prevent the problem from the next boot on, but it has not run on a real boot yet.

The drill read only from the media pool: the HandBrake shares had no file large enough for the test reader, so the TrueNAS path was checked through the watcher’s log and the NFS connection table rather than through reads.