When the Host Dies Hard: Getting Crash Evidence from pstore and the BMC

Purpose

On 2026-10-22 at 19:41:59 local time, the main Proxmox host had a kernel panic during a ZFS scrub. The cause was a fatal PCIe error on an NVMe carrier card. Afterwards the journal and the central log server showed nothing. The journal just stopped at 19:41:59, because a panic never reaches the disk.

A journal that ends cleanly does not mean a clean shutdown. The evidence was in two places nobody was collecting: the pstore/ERST dump the kernel leaves in firmware, and the BMC’s logs. Both now ship to Graylog, and a heartbeat makes “no BMC lines” mean “the shipper is broken” instead of “nothing happened”. The hardware side of this story is in the PCIe switch card post. This post covers how to get the evidence off the host. The central log server is described in the Graylog post.

Deployment

Two paths feed Graylog, plus a check that watches the BMC path.

flowchart LR
    A["Kernel panic"] --> B["pstore/ERST in firmware"]
    B --> C["Next boot: systemd-pstore writes dmesg.txt"]
    C --> D["Vector file source (tagged pstore)"]
    D --> G["Graylog"]
    E["BMC (SEL, EventLog, SMI)"] --> F["Redfish collector on a 10-minute timer<br/>(new entries only, plus heartbeat)"]
    F --> G
    G --> H["Logging-coverage check (6 h window)"]
    H --> I["Finding on health board if zero lines"]

Vector 0.60.0 runs as a host agent (.deb) on the hardware hosts, amd64 and arm64. The BMC collector is a systemd timer on the admin host that runs every 10 minutes and uses a read-only BMC account. It never writes to the BMC.

Configuration

pstore dumps with Vector

systemd-pstore writes the dump at the next boot to /var/lib/systemd/pstore/<id>/dmesg.txt. The Vector file source reads it like this (sanitized from the real config):

include: ["/var/lib/systemd/pstore/*/dmesg.txt"]
read_from: beginning
fingerprint:
  strategy: device_and_inode

read_from: beginning is needed because the dump exists before Vector starts. read_from: end would have skipped it. The fingerprint uses the device and inode because different dumps can share their first line.

The events are tagged source: pstore with the identifier pstore:<id> and a high severity, so a panic stands out in a stream or an alert. The pstore dump says Kernel panic - not syncing: GHES: Fatal hardware error, and 855 dump lines reached Graylog once shipping was set up.

The BMC collector

The collector makes Redfish GET requests against three logs: the SEL and EventLog entries of the manager, and the SMI entries of the system.

GET /redfish/v1/Managers/Self/LogServices/SEL/Entries
GET /redfish/v1/Managers/Self/LogServices/EventLog/Entries
GET /redfish/v1/Systems/Self/LogServices/SMI/Entries

It remembers the last entry id per log and sends only new entries, once each, as GELF with host: <name>-bmc and _log_shipper: bmc-redfish. The first run sent 176 entries.

The three logs saw different things. The SMI log had PCI PERR critical interrupts at the same boot. The SEL had nothing for the panic, and the EventLog had only the power cycle. Collect all three.

The quiet-versus-dead problem

Pstore dumps and BMC logs are event-driven. A healthy BMC with nothing to report looks exactly like a dead shipper, so “missing means a gap” produces false alarms or misses real failures. The fix is a heartbeat: each run writes one bmc-log-ship heartbeat line when at least one log was read.

The BMC path is now checkable. No line in a 6 h window means the shipper or the BMC is down.

The logging-coverage check

The coverage check counts lines per BMC host over a 6 h window. Zero lines becomes a medium finding on the health board. Measured in Graylog on 2026-10-26, the BMC had 95 messages in 7 days and 0 in the last 24 h. That is a healthy, quiet BMC, which is why the heartbeat matters.

The daily digest rates what arrives:

Evidence	Finding
Crash dump (pstore)	critical
Hardware-error kernel lines	warning
BMC Critical or Warning entries	warning
A boot event per host	one per host

Storage

The kernel leaves the dump in firmware (pstore/ERST), and systemd-pstore writes it to /var/lib/systemd/pstore/<id>/dmesg.txt at the next boot. The collector remembers the last entry id for each BMC log. The shipped evidence lives in Graylog.

Volumes in Graylog, measured 2026-10-26:

Stream	Messages
pstore	903 in 30 days
Kernel hardware-error lines	1,285 in 7 days
Boot events	234 in 7 days
BMC	95 in 7 days, 0 in the last 24 h

Access

The BMC account is read-only, and the BMC’s TLS key is pinned. Account details, the BMC address and the credential paths are not part of this post. Graylog is where the evidence is read afterwards.

Notes

• At the next boot after the panic, the firmware replayed error records (BERT/GHES). That gave 314 kernel lines in 24 h, with corrected PCIe errors on the switch card’s downstream ports.
• The outcome: on 2026-10-27 the PCIe-switch NVMe carrier was replaced with a switchless x16 card. Each NVMe now gets its own Gen4 x4 link, and the first full scrub after the swap finished with 0 errors.
• The shipper is still running as a 10-minute timer on the admin host.
• A clean journal end is not proof of a clean shutdown. Check pstore and the BMC.