Post

A second local-path-provisioner: node-local storage for the k3s LXC nodes

A second local-path-provisioner: node-local storage for the k3s LXC nodes

Purpose

When the cluster moved from Raspberry Pi nodes to three x86 LXC nodes, Longhorn could no longer run, and every app’s volume needed a new home. The answer is a second instance of Rancher’s local-path-provisioner, renamed local-zfs. It gives each PVC a directory on a ZFS-backed volume that is mounted into each node container.

It is simple and fast. The cost is that each volume is pinned to one node and the PVC size is a label, not a quota. That shapes cleanup, moves and backups, and it is the reason to read this before choosing between Longhorn, NFS and node-local storage.

Deployment

The provisioner runs in the local-zfs namespace as one replica (1/1 ready), using the image rancher/local-path-provisioner:v0.0.33. Its provisioner name is picluster/local-zfs. An ArgoCD app called local-zfs manages it: Synced and Healthy, with automated sync, prune off and self-heal on.

This is a second instance on purpose. The local-path provisioner bundled with k3s is left untouched, so k3s upgrades don’t interfere with it. Editing k3s’s bundled manifests to reuse the built-in one would not survive an upgrade.

Both the provisioner pod and the helper pods only schedule on nodes labelled kubernetes.io/arch=amd64 and picluster/lxc=true, and they tolerate the picluster/amd64-lxc taint. The three worker nodes are the LXC nodes.

flowchart TD
    A["PVC<br/>class local-zfs"] --> B["Provisioner pod"]
    B --> C["Helper pod<br/>creates a directory"]
    C --> D["ZFS-backed mount<br/>on one worker node"]
    B --> E["PV hostPath<br/>node affinity to that node"]
    E --> F["App pod<br/>pinned to that node"]
    A --> G["PVC deleted"]
    G --> H["PV Released<br/>Retain keeps the data"]
    H --> I["Manual cleanup"]

Configuration

The StorageClass local-zfs is the cluster default:

1
2
3
4
provisioner: picluster/local-zfs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false

It was created on 2026-10-17 as a non-default class and became the default on 2026-10-20.

The local-zfs-config ConfigMap has four parts:

  • nodePathMap with one default path for all nodes (DEFAULT_PATH_FOR_NON_LISTED_NODES), so every node uses the same directory.
  • setup, a mkdir with mode 0777 on the volume directory.
  • teardown, which removes that directory recursively.
  • helperPod.yaml, the helper pod spec. It uses rancher/mirrored-library-busybox:1.36.1.

Because the class uses Retain, teardown never runs when a PVC is deleted.

Storage

Each PV is a hostPath volume with node affinity to one node. The directory is created on whichever node the consuming pod lands on (WaitForFirstConsumer), and the pod stays on that node afterwards. If that node is down, the app waits.

A hostPath volume has no size limit, so the PVC size is a label, not a quota. A volume that fills up fills the shared backing filesystem on that node.

Today there are 25 local-zfs PVs: 23 Bound and 2 Released. Sizes run from 1Gi to 20Gi:

SizePVs
1Gi9
2Gi7
5Gi5
10Gi3
20Gi1

There are also two 1Ti PVs with no storage class.

The directory behind the class lives on a dedicated volume mounted into each worker container, not on the container’s root disk. That volume has changed once without touching the provisioner config. It started as a zvol per node on the NAS over NVMe/TCP, then became local ZFS on the hypervisor after the host migration (see the migration post).

Cleaning up and moving volumes

A Released PV stays around because of Retain. Cleaning it up takes two deliberate steps: delete the PV object, then remove its directory from inside the node container. ArgoCD prune doesn’t change this: deleting a PVC never deletes data.

To move a PV to another node, the directory is copied to the new node and the PV is recreated with the new node in its node affinity. A migrate script in the platform repo does this.

Backups

Backups follow the storage, not Kubernetes. Since 2026-10-29 the dedicated mount of each worker container is included in the container backups. There are two layers:

  • the nightly container backup job (01:30) to the backup server, encrypted at the source;
  • nightly ZFS pulls of the three PV datasets to the offsite NAS (01:30 to 01:40).

There is no per-PVC backup. The old zvol replicas on the NAS belong to the earlier setup and are not the current backup.

Access

Apps don’t talk to the provisioner. They request a PVC without naming a class and get local-zfs, because it is the default. The volume directory is created with mode 0777 by the setup script.

Notes

  • The PVC size is a label. Watch the backing filesystem, not the PVC.
  • WaitForFirstConsumer pins the app to one node, so a node outage means that app waits.
  • With Retain, cleanup is manual and deliberate.
  • A separate instance survives k3s upgrades. Reusing the bundled provisioner would have meant editing files that k3s manages, the same lesson as the CoreDNS .skip file.
  • The trade-off against Longhorn and NFS: this is simple and fast, but each volume is pinned to one node.
This post is licensed under CC BY 4.0 by the author.