Retiring a Proxmox host's guests without losing the rollback

Purpose

This is the order of gates I used to retire the old Proxmox host after the migration to the new one. Nothing on the old host was destroyed until the offsite copy of the backups was proven, and the final backups went into their own protected namespace. Each destructive step ran through a guard script and had its own approval.

Deployment

The old host ran five containers: three k3s nodes, a control-plane node and a Git server. Their replacements had been running on the new host for 7 days with no rollback. The old containers were stopped with autostart off and their volumes intact. That was the rollback until the last gate.

flowchart TD
    A["7 stable days on new host"] --> B["Offsite incremental, GUIDs match"]
    B --> C["Final backup into separate protected namespace"]
    C --> D["PBS verify = ok"]
    D --> E["Guard script: destroy one container at a time"]
    E --> F["Remove storage plumbing"]
    F --> G["Prune NFS exports"]
    G --> H["Delete old replicas offsite"]
    C --> R["Keep until 12-31, then unprotect and prune"]
    A -->|check fails| S["Stop"]
    B -->|check fails| S
    D -->|check fails| S
    E -->|check fails| S

Configuration

Gate 1: the offsite copy. The Proxmox Backup Server (PBS) datastore is replicated to an offsite ZFS pool by a pull from the offsite side (see the restricted pull user and the encrypted receive side). The first full send had finished. The next scheduled run had to be an incremental one whose snapshot GUID matched on both sides. On 2026-11-02 the 05:30 run was an incremental pull, the GUIDs matched and the offsite pool was healthy.

Gate 2: a separate namespace for the final backup. The final backups went into a separate PBS namespace in the same datastore, not into the root. The old and new hosts share container IDs. Archiving at the root would have put the old snapshots in the same groups as the live backups, and “restore the newest” could have picked the wrong one.

A second Proxmox storage entry pointing at that namespace was added on the old host, and the backups ran in stop mode. The extra data-volume mount on the three k3s nodes is not included in backups by default, so I flagged it for inclusion first. All five snapshots were marked protected. They are kept until 2026-12-31, with a reminder to unprotect and prune them then and remove the archive storage entry.

Gate 3: verify. A PBS verify ran on the archive namespace only. The task finished OK with 0 errors, and all five snapshots showed protected and verified. The plan required verify=ok on every snapshot. An earlier dry run stopped on a snapshot whose verify state was none, as it should.

Gate 4: the guard script. The destroy ran as one script with a typed confirmation. Its pre-checks:

• the old and new hosts are who they claim to be
• all five replacements are running on the new host
• every archive snapshot is protected and verified
• each old container is stopped, has the expected hostname, autostart off and no lock

It destroyed containers one at a time, re-checking host, state and name in the same call as each destroy, and stopped at the first failure. The same IDs exist in production on the new host, so a wrong target would have hit a live container.

Every destructive step had its own approval and its own script, each tested on a copy or read-only first.

Storage

The destroy freed space on the old host’s ZFS pool: 559 GB free before, 703 GB after. The three block volumes on the TrueNAS side went from 3 to 0, as did the NVMe-oF namespaces the host saw, and no container datasets were left. The replacement containers were all still running.

Then the plumbing came out, in this order:

1. The Proxmox storage entry for the NVMe-oF volumes, which is only allowed at 0 volumes.
2. The NVMe/TCP disconnect.
3. The locally patched storage plugin, purged.
4. On TrueNAS: the subsystem, host entry, port, API key, service user, custom privilege and group were deleted, and the NVMe-oF service was stopped with autostart off.

Some of the TrueNAS items would not delete through the UI. I removed them with the middleware CLI, matched by name, after exporting the config.

The same pass removed three old replicas of the container volumes from the offsite pool: 28 GB, each with 14 snapshots and no holds, after checking that no unit referenced them. The live replicas stayed.

Access

NFS exports were pruned, but not all of them. The media export kept two clients (the new host and one other) and dropped five powered-off Pis and the k3s node addresses. The Victoria export stayed, because the same node addresses mount it for a live file-browser volume. The checks were exportfs -v, the media mount still writable from the new host, and the Victoria volume still listing files. The exports file is managed by Houston (45Drives’ Cockpit plugin), so saving from its UI can rewrite it.

Notes

• Removing the NVMe-oF subsystem and host entry cleared the standing “accepts any host” warning from the daily ZFS report, along with an NVMe/TCP path nothing used any more. The check is the next day’s report.
• Destroying stopped containers freed no RAM, only defined-but-unused limits. Guest limits are now 45 GiB on 62, so the host is no longer overcommitted.
• On 12-31: unprotect and prune the five archive snapshots, then remove the archive storage entry.