Overcommitting a 64 GB Proxmox host: three crashes and a memory budget

Purpose

The main Proxmox host has 64 GB of RAM, about 62.6 GiB usable. In October 2026 it crashed three times in three days, and last -x showed the boots of 2026-10-21, 10-22 and 10-23 all ending in “crash”. CPU use was 13 to 16% throughout. This post covers how the evidence pointed at memory, what the guest limits added up to, and what changed. One of the three crashes has a different documented cause, covered below.

Deployment

The host runs guests as LXC containers and VMs. The ones that matter here are a storage VM with PCIe passthrough, two Docker hosts, a log server, a git server, an agent host and three k3s nodes. Each guest has a memory limit. The sum of those limits is what turned out to matter.

flowchart LR
    A["Guest limits 101 GiB<br/>+ passthrough VM 24 GiB pinned"] --> C["Host RAM 62.6 GiB exhausted"]
    B["Big jobs start together<br/>(image pull, backup, 27B model load)"] --> C
    C --> D["Hang: pings work, nothing answers"]
    D --> E["Power cycle"]

flowchart LR
    F["Cut limits to measured peaks"] --> G["ZFS ARC max 8 to 4 GiB"]
    G --> H["Move heavy services to a second host"]
    H --> I["Pressure 0.00"]

Configuration

Reading the evidence

last -x | head shows how each boot ended. A “crash” ending means the host went down without a clean shutdown, which separates a hang or reset from a reboot.

The outage on 2026-10-23 at about 06:16 looked like a network fault. Ping and TCP handshakes worked for the host and every guest, but no data came back: no SSH banner, no web UI, no reverse proxy, log server, git server or LLM API. Calls from the agent host to the LLM server kept succeeding during the outage, so the uplink was not simply dead. A power cycle recovered it.

The journal was continuous up to the reset, with no OOM kills, no allocation failures, no hung-task reports and no pstore dump. The Proxmox node statistics froze at 06:16 while the guests’ statistics continued to 06:19. Memory was 58.7 of 62.6 GiB at 06:16, and two heavy jobs had just started together: an app upgrade (docker compose up -d, an image pull plus a recreate) and the LLM server loading a 17 GB model.

The assessment at the time was “not proven, most likely host memory pressure”. The running guests’ limits added up to 101 GiB on a 62.6 GiB host. Container limits are only ceilings, but the storage VM’s 24 GiB is pinned by PCIe passthrough, so it counts as used.

The third crash was about 03:45 on 2026-10-24 and was found at 08:03. On that boot a host-level OOM at 01:31 killed the cluster’s VIP daemon twice. The kernel line for it contained constraint=CONSTRAINT_NONE ... global_oom, which means the host ran out of memory, not a single container. The journal then stopped at 03:45:31 in the middle of routine messages, with no shutdown lines: a hang or reset, not a clean reboot.

Adding up the limits

At 08:03 on 10-24 the host had 53 of 62 GiB used and 8 GiB available. Guest limits came to 91 GiB, about 1.6 times the RAM, plus up to 8 GiB of ZFS ARC (2.8 GiB in use). The largest limits were the storage VM at 24 GiB, two Docker hosts at 12 GiB each, the log server at 8 GiB and three k3s nodes at 7 GiB each.

Cutting to measured peaks

The first cuts, on 10-23, were applied live. Each limit was compared with the 30-day peak of the guest:

Guest	Limit before	Limit after	30-day peak
Agent host	8 GiB	4 GiB	1.5 GiB
Git server	12 GiB	6 GiB	4.2 GiB
Backup server	4 GiB	2 GiB	0.3 GiB

The morning of 10-24 brought a second round, also live, with no guest restarted:

Guest	Limit before	Limit after	Peak
Git server	6 GiB	5 GiB	4.2 GiB (week)
Three k3s nodes	7 GiB each	6 GiB each	3.2 to 3.7 GiB
Intel Docker host	12 GiB	8 GiB	5.5 GiB

The NVIDIA Docker host stayed at 12 GiB against a peak of 10.8 GiB. Guest ceilings went from 91 to 84 GiB. Together with the ARC cut from 8 to 4 GiB, that is about 11 GiB less claimable at once. The limit of a container can be changed on the running guest:

pct set <id> -memory <MiB>

The k3s nodes are LXC containers, and the service there is k3s-node, not k3s-agent. Restart it after a cut so the node reports the new allocatable memory. After the cut the nodes reported 6Gi, Ready, with 0 container restarts.

The changes were also applied in Terraform (PR #11 merged), and plan on main printed “No changes”.

Storage

The ZFS ARC is the other claim on RAM. Its maximum went from 8 to 4 GiB, set live, in a modprobe option and in the initramfs so it survives a reboot:

echo $((4*1024**3)) > /sys/module/zfs/parameters/zfs_arc_max
# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=4294967296
update-initramfs -u

The 4 GiB cap is still in place.

Access

The host-wide memory exhaustion hit several times between 2026-10-21 and 10-25. It usually came when a large local LLM (27B) loaded while the guests were already near the ceiling. On 10-25 the k3s cluster and the core ops services moved to a second host. The epyc-to-pemba migration post tells that story, so it is not repeated here.

On 2026-10-26 the old host ran only the storage VM and the two Docker hosts. Memory was 47 of 62 GiB used with 15 GiB available, and /proc/pressure/memory read 0.00 across avg10, avg60 and avg300:

cat /proc/pressure/memory

No further crashes are recorded after 10-24.

Notes

• Memory starvation looks like a network fault. Pings and TCP handshakes kept working while nothing answered.
• Count passthrough VM memory as used. Container limits are ceilings, but a passthrough VM’s RAM is pinned.
• A hang can leave no trace. The 10-23 journal had no OOM lines. Per-minute pressure history (sysstat or PSI) is the missing evidence, and it is still on the list of optional items, along with kdump or netconsole so a hang leaves something behind, and more RAM with a CPU upgrade.
• Not every crash in that window was memory. The 10-22 crash was a kernel panic from a PCIe error on an NVMe carrier card, written up in the crash evidence post.