Moving k3s and core services from epyc to pemba

Purpose

The Proxmox host epyc (EPYC 8024P, 8 cores / 16 threads, 64 GiB RAM) ran the k3s cluster, Graylog, Hermes, Gitea and more. It hit host-wide memory exhaustion several times between 2026-10-21 and 2026-10-25, usually when a large local LLM (a 27B Qwen model) loaded while the guests were already close to the ceiling. On 2026-10-25 the k3s cluster and the core ops services moved to a second host, pemba (i5-12600H, 16 threads, 94 GiB RAM, NVMe rpool). This post records what moved and how.

Deployment

Before and after:

flowchart LR
    A["epyc<br/>64 GiB"] -->|"k3s nodes via PBS restore"| B["pemba<br/>94 GiB"]
    A -->|"Graylog, Hermes, Gitea CTs"| B
    A -->|"PV data copy over 10 GbE"| B

What moved to pemba:

• k3s cluster: the four LXC nodes (one control plane, three workers) were restored from Proxmox Backup Server onto pemba’s local ZFS.
• Graylog, Hermes and Gitea (CTs 107, 108 and 109): each got a pinned static address and was checked end to end: log ingest, the agent dashboard and webhooks, Git over HTTPS and SSH, the CI runner, and ArgoCD pulling from Gitea. See the Graylog post and the Hermes agent post for what those services do.

State after the move (kubectl, 2026-10-26): 4 of 4 nodes Ready (Kubernetes v1.30.2+k3s2), 66 pods Running, and 42 ArgoCD applications Synced and Healthy.

Configuration

The cutover went in this order:

1. Graylog and Hermes moved first, which freed enough memory on epyc to restore the cluster on pemba.
2. The old copies on epyc stayed stopped, so rolling back meant starting them again.
3. The epyc to pemba link was moved to a 10 GbE SFP path for the cutover (about 8.8 Gbit/s aggregate measured), so the persistent-volume copy ran host to host.

No TrueNAS reboot was needed.

Storage

The persistent volumes were copied off the old TrueNAS NVMe/TCP zvols onto local ZFS. The cluster’s default StorageClass is now local-zfs, and the PVs are local-zfs volumes (1 GiB to 20 GiB each). The only exceptions are two 1 TiB NFS volumes, which the media file browsers use. k3s no longer depends on TrueNAS. /media_pool is served to the cluster over NFS from the storage host.

Nightly PBS backups now run on pemba, staggered between 00:30 and 03:00.

Access

Each moved container (Graylog, Hermes, Gitea) has a pinned static address, and the checks listed under Deployment confirmed they work end to end after the move.

Notes

• Monitoring: the ops digest and health board found the new host by themselves and added a pve-pemba collector with no config change. See the ops digest post.
• Worker memory: after a couple of OOM kills under pemba’s tighter limits, worker memory went from 6 to 8 GiB.
• Open point: the old epyc guests are stopped, not deleted. Remove them once the new layout has proved stable.
• Tracking: k3s-platform milestone 14 (#92) and homelab-ops #159, #160 and #161.