Kafka

Kafka

Purpose

A single-broker Apache Kafka cluster running in KRaft mode (no Zookeeper), plus kafka-ui for browsing topics/messages. Both run in the default namespace of the k3s cluster and are managed by ArgoCD from the k3s branch of the kafka Gitea repo.

Broker

Item	Value
Image	apache/kafka:4.4.0
Replicas	1 (strategy: Recreate — required by the ReadWriteOnce PVC)
Mode	KRaft combined mode: one node acts as both broker and controller (KAFKA_PROCESS_ROLES=broker,controller)
Ports	9092 (PLAINTEXT, in-cluster), 9093 (CONTROLLER, KRaft consensus), 9094 (EXTERNAL, LAN clients)
Storage	PVC kafka-data-zfs, 2Gi, storageClassName: local-zfs → /var/lib/kafka/data
Resources	requests 250m/512Mi, limits 1000m/1536Mi

Replication factor is 1 throughout (offsets/transaction-state-log) — appropriate for a single-broker homelab setup, not a real multi-node quorum.

Services

• kafka — ClusterIP, ports 9092/9093, DNS kafka.default.svc.cluster.local
• kafka-external — LoadBalancer (MetalLB), port 9094, for LAN clients outside the cluster. Split into its own Service so the internal ClusterIP service doesn’t also carry a MetalLB-assigned external IP.

KAFKA_ADVERTISED_LISTENERS’s EXTERNAL entry hardcodes the expected MetalLB IP (192.0.2.35). If MetalLB ever assigns kafka-external a different address, external clients will be told to reconnect to a stale IP — check kubectl get svc kafka-external if external producers/consumers stop connecting.

kafka-ui

Item	Value
Image	provectuslabs/kafka-ui:v0.7.2
Service	kafka-ui, ClusterIP, port 8080
Bootstrap servers	kafka.default.svc.cluster.local:9092

Ingress

Traefik IngressRoute for kafka-ui only (the broker’s wire protocol is raw TCP, not HTTP, so it’s reached via the LoadBalancer Service instead of an ingress route):

• kafka.prod-k3s.homelab.example (middleware: default-headers)

TLS via the shared local-example-com-tls wildcard cert.

Storage risk

kafka-data-zfs is a node-local local-zfs volume: a directory on that node’s ZFS-backed volume (a zvol on the TrueNAS NVMe pool, attached to the k3s LXC node), so the pod is scheduled where its volume is. A single broker on one node’s volume has no in-cluster redundancy: if that node is down, the broker waits for it. The topic data is covered by the node volume’s nightly snapshots, not by Kafka replication.

GitOps

Managed by ArgoCD (Application in argocd/apps/kafka.yaml), watching the k3s branch of gitea.git.homelab.example/bjones/kafka with prune: true and selfHeal: true — changes are made by committing to that branch, not by running kubectl apply by hand. Both images are pinned to :latest, so Renovate can only track digest changes here, not real version bumps.