From "the host feels slow" to one process: attributing Proxmox and ZFS I/O delay to a guest

Symptom

The Proxmox host summary graph showed steady I/O delay. It was the kind that makes everything feel sluggish without pointing at a culprit. This post is the recipe I used to go from that graph to a single process, top-down: confirm the delay is real, find the busy pool, find the guest, then name the process. Everything was read-only and nothing had to be restarted.

The numbers below were measured on 2026-10-26, in two sampling passes about five minutes apart with nothing changed on the host. They can’t be re-measured for that day, so they are published as measured then. A re-check on 2026-10-30 is at the end.

flowchart TD
    A["I/O delay in the host summary graph"]
    B["Is it real?<br/>PSI + iostat"]
    C["Which pool?<br/>zpool iostat (IOPS vs MB/s)"]
    D["Which guest?<br/>cgroup io.stat deltas,<br/>devices mapped to vdevs"]
    E["Which process?<br/>per-PID read_bytes in the guest"]
    F["Culprit: embeddings worker<br/>on the QLC pool"]
    G["High reads but pool idle<br/>at the block layer"]
    H["Served from ARC,<br/>not the culprit"]
    N["Note: ARC pruned under<br/>memory pressure"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    E --> G
    G --> H
    N -.-> F

Step 1: is it real?

Start with the kernel’s own pressure numbers and a short iostat run.

cat /proc/pressure/io
iostat -x 5 3

On 10-26 the I/O pressure “some” line read avg10 about 6.2 and avg300 about 8.3. %iowait was 3.8 to 4.1 and load was about 2.7. That is moderate but steady, which is what the graph showed. To see the per-minute iowait history from Proxmox itself:

pvesh get /nodes/<node>/rrddata --timeframe hour

Step 2: which pool?

zpool iostat -v 5 3

The QLC SSD pool, three 2 TB QLC SATA SSDs in one raidz1 vdev, did about 2,390 read IOPS at only about 10 MB/s. Many operations with little bandwidth means small random reads. The HDD media pool, the backup pool and the rest were idle at the block layer.

Step 3: which guest?

Each LXC has its own cgroup with an io.stat file. Read it twice, 10 seconds apart, for every running container, and diff rios and rbytes per device.

cat /sys/fs/cgroup/lxc/<id>/io.stat
sleep 10
cat /sys/fs/cgroup/lxc/<id>/io.stat

io.stat is per block device, listed by major:minor. The diff only means something once you map the hot devices back to the pool’s member disks with lsblk. Here about 2,060 random read IOPS belonged to the Docker LXC, reading the three QLC disks directly. Other guests were negligible. Containers that had been stopped the day before, when they moved to another host, could be ruled out at once.

Step 4: which process?

Enter the guest and read /proc/<pid>/io twice for every process, then sort by the delta.

pct enter <id>
cat /proc/<pid>/io

Look at read_bytes, not rchar. read_bytes counts only reads that reach the block layer, while rchar counts cached reads too. In the Docker LXC the top reader was Frigate’s semantic-search embeddings worker (frigate.embeddings) at about 18 MB/s. It had the Frigate database and its WAL open while it scanned the recordings dataset. Frigate runs from the stable-tensorrt image with restart policy unless-stopped, and its recordings sit on a host bind mount from the QLC pool. The setup is described in the Frigate post.

The red herring

Jellyfin also showed high process-level reads. A process with big reads is not automatically the disk culprit, so check the pool it reads from at the block layer. The media pool was idle at the disk, which means those reads were served from ARC, the ZFS cache in RAM.

Why it hurt

Two things stacked up. The first is the workload: small random reads on the slowest pool. Earlier measurements on the same pool gave about 26 ms read and 29 ms write latency, against about 1 ms on an NVMe pool.

The second is memory. The host had 48.4 of 64 GiB used and 5.3 GiB free, and the arc_prune kernel thread was pinned at 100 % CPU, so the ARC was being shrunk. Reads that would have been cached went to the slow disks. You can check this with cat /proc/spl/kstat/zfs/arcstats (size, c, c_min) and top. The ARC max had been cut from 8 to 4 GiB on 2026-10-24 after memory exhaustion on the host, and the k3s cluster and core services moved to a second host on 2026-10-25, as told in the migration post.

Where it stands on 2026-10-30

Memory pressure made it worse on 10-26, but freeing RAM alone did not fix it while the ARC stayed capped. On 10-30 about 32 GB was available and arc_prune was gone from top. The ARC was still capped at 4 GiB, so the freed RAM could not reach the cache.

The load turned out to come in bursts. Samples that day showed about 2.35k reads/s at one moment, about 1 a few minutes later and about 1.6k soon after, which fits a background worker that runs in bursts. During a burst the same method gave the same answer: the per-cgroup io.pressure was all in the Docker LXC, the top read_bytes reader was the embeddings worker at about 13.5 MB/s, and the disks’ own read latency was low, about 0.2 to 0.3 ms.

The problem is still present as of 2026-10-30. No fix has been applied.

Options

• Raise the ARC cap so the free RAM can hold more of the working set.
• Tune Frigate’s semantic search, the workload itself.
• Move the recordings to faster storage. The move is planned, with no date yet.

A follow-up post will have before and after numbers once the storage move is done.