Kafka

Kafka

Purpose

A single-broker Apache Kafka cluster running in KRaft mode (no Zookeeper), plus kafka-ui for browsing topics/messages. Both run in the default namespace of the k3s cluster and are managed by ArgoCD from the k3s branch of the kafka Gitea repo.

Broker

Item	Value
Image	apache/kafka:latest
Replicas	1 (strategy: Recreate — required by the ReadWriteOnce PVC)
Mode	KRaft combined mode: one node acts as both broker and controller (KAFKA_PROCESS_ROLES=broker,controller)
Ports	9092 (PLAINTEXT, in-cluster), 9093 (CONTROLLER, KRaft consensus), 9094 (EXTERNAL, LAN clients)
Storage	PVC kafka-data, 10Gi, storageClassName: local-path → /var/lib/kafka/data
Resources	requests 250m/512Mi, limits 1000m/1536Mi

Replication factor is 1 throughout (offsets/transaction-state-log) — appropriate for a single-broker homelab setup, not a real multi-node quorum.

Services

• kafka — ClusterIP, ports 9092/9093, DNS kafka.default.svc.cluster.local
• kafka-external — LoadBalancer (MetalLB), port 9094, for LAN clients outside the cluster. Split into its own Service so the internal ClusterIP service doesn’t also carry a MetalLB-assigned external IP.

KAFKA_ADVERTISED_LISTENERS’s EXTERNAL entry hardcodes the expected MetalLB IP (192.168.1.35). If MetalLB ever assigns kafka-external a different address, external clients will be told to reconnect to a stale IP — check kubectl get svc kafka-external if external producers/consumers stop connecting.

kafka-ui

Item	Value
Image	provectuslabs/kafka-ui:latest
Service	kafka-ui, ClusterIP, port 8080
Bootstrap servers	kafka.default.svc.cluster.local:9092

Ingress

Traefik IngressRoute for kafka-ui only (the broker’s wire protocol is raw TCP, not HTTP, so it’s reached via the LoadBalancer Service instead of an ingress route):

• kafka.prod-k3s.iot-proxmox.net (middleware: default-headers)
• www.kafka.prod-k3s.iot-proxmox.net

TLS via the shared local-example-com-tls wildcard cert.

Storage risk

kafka-data uses storageClassName: local-path with no nodeSelector pinning the broker pod to a specific node. Unlike Longhorn-backed PVCs (replicated across all 3 nodes), a local-path volume is tied to whichever node it was first created on — if that node or the pod is lost, both the running broker and its on-disk topic data go with it, with no in-cluster redundancy.

GitOps

Managed by ArgoCD (Application in argocd/apps/kafka.yaml), watching the k3s branch of gitea.prod.iot-proxmox.net/bjones/kafka with prune: true and selfHeal: true — changes are made by committing to that branch, not by running kubectl apply by hand. Both images are pinned to :latest, so Renovate can only track digest changes here, not real version bumps.