How an empty mountpoint filled the Proxmox root pool

Purpose

A retired storage pool left an ordinary directory behind, and a scheduled copy job wrote 1.6 TB into it overnight. The Proxmox root pool reached 96 % with 0 B available, and that one full disk caused three separate failures. This post walks through how to find the problem, how to recover, and the one-line guard that makes an empty mountpoint fail loudly. It is written for anyone who moves pools between hosts.

Deployment

The pieces involved:

• A Proxmox host whose root pool also holds the root disks of two containers.
• A NAS appliance with a backup pool and a scheduler for replication tasks.
• A storage VM that took over a storage pool from the Proxmox host. The storage VM is covered in the TrueNAS VM post.
• A git server container with a separate Postgres container, and an NVR that records to a dataset on an SSD pool.

flowchart TD
    A["Pool moves to the storage VM"]
    B["Mountpoint directory left behind"]
    C["Nameless nightly rsync task still enabled"]
    D["1.6 TB written to the root pool"]
    E["Root pool 96 percent full, 0 B free"]
    F["pmxcfs full, cfs-lock timeouts, UI 401"]
    G["Restart policy drift, Postgres stays down"]
    H["Readonly camera dataset, NVR cannot record"]
    I["chattr +i on the empty directory<br/>missing mount makes writes fail"]
    A --> B
    B --> D
    C --> D
    D --> E
    E --> F
    E --> G
    E --> H
    I -.-> B

Configuration

The cause

A nightly rsync replication task on the NAS appliance had been created with a blank name. It was enabled since 2026-09-22 and ran daily at 00:00. It pushed a 1.6 TB camera-footage dataset from the backup pool to the old pool’s path on the Proxmox host.

On 2026-10-14 at 18:38 the old pool’s mount was torn down when the pool moved to the storage VM. The directory stayed, so the path became an ordinary directory on the root pool. The task ran at midnight as usual and about 1.6 TB landed on the root pool overnight. Because the task had no name, it was easy to miss in the task list when the pool was retired.

Finding the space

On 2026-10-15 the root pool had 0 B available to every dataset on it: the host’s own root and the two containers’ root disks. Start with the filesystem, then walk down. -x keeps du on one filesystem so other mounts don’t distort the numbers.

df -h /
du -xh --max-depth=2 /

A large directory that should have been a mountpoint stands out. To check whether a path is a mount right now:

findmnt <path>

No output means the path is a plain directory. To find scheduled tasks still pointing at a retired pool, filter the timers for the NAS scheduler’s units:

systemctl list-timers

Failure 1: the cluster filesystem

The Proxmox cluster filesystem (pmxcfs) logged “database or disk is full”, every cfs-lock timed out, and the web UI returned 401 for all logins. A pvedaemon log line like this, together with a UI 401, means pmxcfs is wedged:

cfs-lock ... got lock request timeout

Here the 401 came from a full disk, not from authentication. The first “database or disk is full” was logged at 07:56 on 2026-10-15 and the services were restarted at 17:08 the same day, so pmxcfs was stuck for about 9 hours. The journal does not show when anyone first hit the 401.

Recovery starts with freeing the space. On a single-node host, once there is room, check the cluster database and restart the services:

sqlite3 <cluster-database> "PRAGMA integrity_check;"
systemctl restart pve-cluster pvedaemon pveproxy pvestatd pvescheduler

Failure 2: restart policy drift

After the git server container was rebooted, its Postgres container stayed down. The live container had restart policy no, while its compose file said unless-stopped. The drift came from a hand recreate days earlier. Compare the live policy with the compose file, then fix the live one:

docker inspect -f '' <ctr>
docker update --restart=unless-stopped <ctr>

Failure 3: a read-only dataset

Found in the same check: the camera dataset on the SSD pool had been left readonly=on by a migration receive, so the NVR couldn’t record.

zfs get readonly <dataset>
zfs set readonly=off <dataset>

The guard

With the mount stopped, make the empty directory immutable:

chattr +i <mountpoint>

Writes to the unmounted path then fail instead of filling the parent filesystem, and mounting over it still works. The guard was later used on a new host: the empty mountpoint for an NFS media mount was made immutable, and the NFS mount still mounts over it. It was tested when added: writes failed and the mount worked.

Storage

The root pool was the victim. Once it hit 96 % with 0 B available, everything on it was affected: the Proxmox root, pmxcfs and the two containers’ root disks. The 1.6 TB camera dataset itself lives on the NAS backup pool, and the NVR records to a dataset on the SSD pool.

Access

The visible symptom was an access one: the Proxmox web UI returned 401 for all logins while the disk was full. Nothing was wrong with authentication.

Notes

• The guard went only on the new host’s NFS mountpoint. The old pool path on the Proxmox host got no immutable flag. Nothing writes there any more, because the nameless task was retired.
• Retiring a pool means retiring every job that writes to its path, including a task with no name.
• A Proxmox UI 401 can be a full disk.
• One full disk exposed two older, unrelated drifts: the restart policy and the readonly flag. After any outage, check what changed on the way back up.
• An empty mountpoint is a silent trap. Make it immutable so a missing mount fails loudly.

Checklist for retiring a pool:

1. List every scheduled task, replication job and backup that names the pool’s path, including unnamed ones.
2. Disable or delete them before the mount goes away.
3. Run findmnt <path> afterward and confirm the directory is empty.
4. Run chattr +i <path> on the empty directory if a mount might return.
5. Check df -h / the next morning.