Centralized Logging with Graylog

Purpose

A central log collector for the whole lab. Graylog ingests every source — Kubernetes and Docker hosts, reverse proxies, media-management tools, GPU workloads — into one searchable store with per-app streams and dashboards that pivot on host labels rather than hardcoded widgets.

Graylog runs as a native install (server + data node) in its own LXC container on the Proxmox host, not Docker. Two inputs receive everything: GELF UDP for Docker’s native gelf log driver and short stdout lines from voice containers and Immich, and GELF TCP with null-byte framing for all file logs shipped by Vector sidecars.

Architecture

flowchart LR
  subgraph Sources
    T1[Traefik<br>k3s]
    T2[Traefik<br>Docker #1]
    T3[Traefik<br>Docker #2]
    T4[Traefik<br>Docker #3]
    A[Arr stack<br>5 apps + gluetun VPN]
    V[Voice containers<br>Ollama + Wyoming]
    IM[Immich]
  end

  subgraph Shippers
    VC1[Vector sidecar]
    VC2[Vector sidecar]
    VC3[Vector sidecar]
    VC4[Vector sidecar]
    VS[Vector in arr pod]
    GD[Docker gelf driver]
  end

  subgraph Graylog
    IN["GELF TCP"]
    GU["GELF UDP"]
    J[on-disk journal]
    S[streams +<br>Immich pipeline]
    D[dashboards]
  end

  T1 --> VC1
  T2 --> VC2
  T3 --> VC3
  T4 --> VC4
  A  --> VS
  V  --> GD
  IM --> GD

  VC1 & VC2 & VC3 & VC4 & VS --> IN
  GD --> GU
  IN & GU --> J --> S --> D

Traefik logging on four environments

Traefik writes access.log and traefik.log to a shared volume in each environment — Docker hosts or the k3s cluster — and a Vector sidecar tails those files, enriches them with node_name, then ships GELF over TCP.

On the k3s cluster, this is the Traefik Deployment in the traefik namespace: the main container (traefik:v3.7.10) shares an emptyDir (traefik-logs) with a Vector sidecar (timberio/vector:0.49.0-debian). The Vector config comes from the traefik-vector-config ConfigMap, mapped at /etc/vector/vector.yaml.

On each Docker host, Vector runs as its own compose service in the Traefik stack (see existing Traefik post). The node_name field is set per-host so dashboards pivot by environment — add a new host and it appears in every widget with no changes needed.

Arr stack logging

The Arr stack runs qBittorrent, Sonarr, Radarr, Prowlarr, Profilarr, and Gluetun inside a single Pod. A vector container in the same pod mounts each app’s config volume read-only at a second mount point and tails the current log file — leaving the apps’ own logging paths untouched.

The Vector config (arr-stack-vector-config) defines one file source per app, all with read_from: end, and a separate remap transform per app to parse level, logger, and message before forwarding GELF TCP to Graylog. Parsing happens in Vector’s VRL remaps — the apps’ log formats vary (pipe-delimited for Sonarr/Radarr/Prowlarr, JSON for Profilarr, bracket-code prefix for qBittorrent), so each gets its own parser.

Host tagging

Every message carries a node_name field set in Vector’s remap — or via an environment variable and Docker log-opt for the GELF driver on containers with short log lines like Ollama, the Wyoming voice pipeline, and Immich. All Docker hosts run the same Traefik template with identical router names, so without the label there is no way to tell one environment’s logs from another.

Streams and pipelines

Graylog has per-app streams plus umbrella streams for Traefik, Home Assistant voice, Arr stack, and Immich. The Immich stream runs a processing pipeline with staged rules: drop stray extractor fields, parse per container, handle stack traces, then normalize level names.

Deployment

Role	Where	Image / version	Management
Graylog	Proxmox LXC	native install (server + data node)	brief
Vector (Traefik sidecar)	k3s namespace traefik	timberio/vector:0.49.0-debian	Helm chart additionalContainers / ConfigMap
Vector (Docker Traefik)	each Docker host	same	compose service
Vector (arr stack)	k3s Pod in default	same	bundled with gluetun Deployment
GELF UDP driver	voice, Immich hosts	built into Docker Engine	compose log-driver: gelf

Configuration

Example from the Traefik sidecar ConfigMap — a file source per log, a remap to set the node label, and a TCP socket sink using the GELF codec with null-byte delimiters:

sources:
  traefik_log:
    type: file
    include:
      - /var/log/traefik/traefik.log

transforms:
  set_host:
    type: remap
    source: .node_name = "k3s"

sinks:
  graylog:
    type: socket
    mode: tcp
    encoding:
      codec: gelf
    framing:
      method: character_delimited
      character_delimited:
        delimiter: "\u0000"

The Arr stack ConfigMap multiplies this pattern: five file sources (one per app log) and five remap transforms, each with a regex or JSON parse tailored to the specific app’s format.

Known limitations

• Docker’s gelf driver ignores the container hostname for source fields — the GELF source is the Docker daemon’s hostname, not the container name. Use the auto-added container_name field instead.
• Graylog-side changes (new streams or extractors) take one to three minutes to affect live traffic, and API-created streams may need an explicit resume.

---

Lessons learned

The 37-million-message backlog

Onboarding a new Docker host gave Vector a two-month-old access.log it had never seen before. No checkpoint, so it started at byte zero — ~37 million messages in one burst. Graylog’s 5 GB on-disk journal hit 95% capacity and began discarding uncommitted entries, which means some real log data was permanently lost. Messages from other sources appeared ten minutes late because the backlog held up its turn in the journal, briefly looking like a broken pipeline across the board.

Fix: every file source now uses read_from: end by default. If you genuinely need to backfill old files, rotate or truncate first. A later test confirmed it — when another sidecar discovered 24 historical daily log files in its directory, it ingested none of them.

GELF’s level field is reserved and must be numeric

Naming a parsed text severity value level — for example “Info” or “Warning” — makes Vector’s GELF encoder reject the entire event. The message was silently dropped with no error surfaced on the Graylog side, so it looked like logs were just vanishing mid-pipeline.

Fix: call that field log_level. It is used consistently in all remap transforms now.

The VPN swallowed the log shipper

In the arr-stack Pod, Gluetun routes all pod egress through the VPN tunnel — including Vector’s own TCP connection to Graylog on the LAN side. Messages never left the tunnel interface heading toward a non-internet address, which means nothing arrived at Graylog and there was no connection error visible anywhere because the packet hit NAT rules instead.

Fix: allow just Graylog’s single internal address in Gluetun’s FIREWALL_OUTBOUND_SUBNETS setting. Reloader restarted the Pod automatically when the Secret changed. Rule for future: any new sidecar added to that Pod needs its destination allow-listed first, before deploying.

GELF over UDP drops big lines

Single log lines exceeding 90 KB were silently dropped when sent over UDP — message too long, packet reassembly fails. The loss was invisible because only certain verbose containers hit that size threshold, and no error logged anywhere in the pipeline.

Fix: use GELF over TCP with null-byte delimiters for any source that might produce large messages. Docker’s native gelf driver on short-line containers (voice, Immich) still uses UDP safely — those outputs never approach the limit.

Unanchored and unconditioned extractors

A Graylog grok extractor without a leading ^ anchor matched a timestamp embedded inside an unrelated log line, producing false-field noise. Another extractor with no condition turned any timestamp-prefixed line — including Postgres connection logs — into an unparseable date field, causing the entire message to be rejected at indexing time and visible only in the indexer-failures view.

Fix: anchor patterns with ^, add conditions to extractors, or handle the stray field in a dedicated pipeline stage before it reaches the extractor chain.

Docker’s gelf driver ignores the container hostname

The GELF specification sets source to the daemon host. Setting hostname: inline in compose overrides just a metadata label — not the field Graylog reads for source routing. So all gelf-driver logs share the same source value regardless of which named container they came from.

Fix: use the auto-added container_name field.

Notes

• Home Assistant Core’s own log file was attempted as a Vector source and rolled back. Vector 0.49’s file source stops watching after HA rotates its log on restart — a known upstream issue — and there was an additional, unexplained delivery stall during testing. The likely future path is a DaemonSet at the node level tailing container logs directly instead of an in-Pod sidecar model; see existing Home Assistant post for deployment context.
• Vector’s checkpoint emptyDir resets on Pod replacement, which is harmless here because the Traefik log volume is also an emptyDir (a new Pod starts with empty logs), and the arr-stack sidecar uses read_from: end.