Prometheus

Prometheus

Cluster-monitoring Prometheus, deployed to the default namespace of the k3s cluster alongside the rest of the homelab’s apps. It scrapes the Kubernetes API server, kubelet node metrics, cAdvisor container metrics, and any Service/Pod annotated for scraping, then feeds the existing Grafana deployment via file-based provisioning.

Deployment

Item	Value
Namespace	default
Image	prom/prometheus:latest
Port	9090
Ingress	prometheus.prod-k3s.iot-proxmox.net (Traefik IngressRoute, TLS)
Storage	PVC prometheus-data, 10Gi, local-path
Retention	15 days (--storage.tsdb.retention.time=15d)
ServiceAccount	prometheus (ClusterRole: read nodes/services/endpoints/pods + /metrics)

reloader.stakater.com/auto: "true" is set on the Deployment, so it restarts automatically when its ConfigMap changes.

Scrape Targets

Configured via a ConfigMap-mounted prometheus.yml:

• kubernetes-apiservers — the API server itself, via the in-cluster service account token.
• kubernetes-nodes — kubelet /metrics on every node, proxied through the API server.
• kubernetes-nodes-cadvisor — per-node container metrics via the kubelet’s cAdvisor proxy; this is what the Grafana dashboard below is built on.
• kubernetes-service-endpoints / kubernetes-pods — any Service or Pod carrying prometheus.io/scrape: "true" (and optional prometheus.io/path / prometheus.io/port) annotations is picked up automatically.

No kube-state-metrics is deployed — the cAdvisor scrape alone is enough for the dashboard in use.

Grafana Integration

Three ConfigMaps are mounted into the existing Grafana deployment (not a separate Grafana install):

ConfigMap	Mounted at	Purpose
grafana-datasource-prometheus	/etc/grafana/provisioning/datasources	Registers Prometheus (http://prometheus.default.svc.cluster.local:9090) as the default datasource
grafana-dashboard-provider	/etc/grafana/provisioning/dashboards	Tells Grafana to load any dashboard JSON dropped into /var/lib/grafana/dashboards into a “Kubernetes” folder
grafana-dashboard-315	/var/lib/grafana/dashboards	The actual dashboard JSON

The dashboard is grafana.com dashboard 315 (“Kubernetes cluster monitoring via Prometheus”), with two local patches:

• ${DS_PROMETHEUS} replaced with the literal datasource name Prometheus, since file-based provisioning skips the UI import’s __inputs substitution.
• The filesystem-usage panels’ device filter changed from the dashboard default (^/dev/[sv]da\d+$) to ^/dev/nvme0n1p2$ — the actual root device on all three cluster nodes (cm4-left, cm4-middle, cm4-right).

Notes

• If a node with a different root device is added to the cluster, the container_fs_* queries in the dashboard JSON need widening (e.g. to ^/dev/(nvme\d+n\d+p\d+|[sv]da\d+)$) and the dashboard ConfigMap re-applied.
• Retention (15d) and the PVC size (10Gi) should be revisited together if the cluster’s metric cardinality grows.