Offsite ZFS over Tailscale with a locked-down pull user
Purpose
An offsite NAS pulls nightly ZFS replicas from the main Proxmox host over Tailscale. It used to log in as root. It now logs in as an unprivileged zfspull user whose SSH key can run one small gate program and nothing else. The copies land encrypted on the offsite pool, and the gate refuses anything beyond the few zfs commands a pull needs.
This post covers the design and two failures along the way: reading files on a replica broke incrementals, and retention deleted the last common snapshot, which forced a resend of about 13.5 TB. The story has a finished ending: holds fixed it, and the first scheduled run after the resend was incremental again.
Deployment
Replication runs as 45Drives Houston tasks, an open-source Cockpit plugin. Each task is a systemd service with an environment file, a JSON definition and a timer. The offsite box runs the pull side, daily. All 13 offsite pull tasks log in as zfspull. The tasks whose source is the second Proxmox host now reach it over Tailscale too (they used the LAN before), so the offsite box can leave the LAN without breaking them.
flowchart LR
A["Offsite NAS<br/>timer starts pull task"] --> B["SSH over Tailscale<br/>zfspull key"]
B --> C["Source host sshd"]
C --> D["Forced-command gate"]
D --> E["Allowed zfs subcommand<br/>on source pools"]
D --> F["Refused: logged to syslog"]
E --> G["zfs send stream"]
G --> H["zfs recv into encryption root<br/>offsite pool"]
Configuration
The source side has two layers.
First, zfs allow delegates destroy,hold,mount,release,send,snapshot,userprop on the source pools to the pull user. Send needs no root, and the login shell is never used.
Second, the key’s authorized_keys line carries three options:
restrict: no pty, no forwarding.from=: only the offsite box’s tailnet address.command=: the gate.
The key is a dedicated ed25519 key, and the offsite ssh config has a Match user zfspull block that uses it.
The gate is a small Python forced command. No shell starts: it runs shlex.split on the original command and then execv of zfs. What it allows:
| Command | Allowed form |
|---|---|
zfs list, get, holds | read-only queries |
zfs snapshot | [-r] dataset@name |
zfs send | including -n estimates, -i/-I, and resume with -t |
zfs hold, release | tag dataset@snapshot |
zfs set | user properties on snapshots only |
zfs destroy | one snapshot at a time |
true | alone, as the connectivity probe |
Everything else is refused and logged to syslog at auth.warning. The shape of the check, as pseudocode:
1
2
3
4
5
def check(argv):
# argv[0] must be "zfs", argv[1] a subcommand in the allowed table
# every dataset argument must sit under an allowed pool
# destroy: exactly one argument, and it must be a snapshot (contains "@")
# return (allowed, reason)
check() is a pure function and is unit-tested. I verified that a shell, cat, destroying a dataset, touching other pools and recv are all refused. The gate replaced root login: the last unrestricted root key from the offsite box on the main Proxmox host had gone unused for 14 days (every login from the box was zfspull), and it has been removed. A root login from the offsite box is now refused.
Storage
The offsite pool has an encryption root, enc, using aes-256-gcm with a hex key. The key sits in a root-only file that a small unit loads at boot. readonly=on and atime=off are inherited by the children. Under it live the media, the camera recordings, the family file shares, the Kubernetes volumes and the backup server’s datastore.
Houston sends non-raw streams. A full receive into a new child of the encryption root is therefore written encrypted with the parent’s key, and each task only needs destDataset_dataset=enc/<ds>.
Retention is 2 weeks on both sides for the offsite tasks. The backup tier keeps 8 weeks and the camera recordings 1 week. The offsite pool reports 0 errors in the daily facts. Other ZFS posts cover different problems: checksum errors from a PCIe switch card and attributing ZFS I/O delay.
Access
The only way in is the pinned key, from the offsite tailnet address, through the gate. Nothing on the source host lets that key open a shell.
Notes
Failure 1: reading a replica breaks incrementals. With atime=on, reading files on a replica counts as a modification. The next incremental receive without -F refused with “destination has been modified”, even though zfs diff showed 0 changed paths. The fix was three commands: zfs rollback <replica>@<newest>, zfs set readonly=on <replica>, and zfs set atime=off <pool>.
Restart=on-failure on the replication unit turned that refusal into a loop every ten seconds, and each try made a new source snapshot: 9 stray snapshots in 90 seconds on one task. Another task crash-looped about 22,500 times before it was disabled.
Failure 2: the common snapshot aged out. Retention kept running on the source while the media task was disabled. The media-* snapshots expired and no common GUID was left. There was no room for old and new copies side by side, so the old copy was destroyed first, and there was no offsite media copy for about 4 days.
The resend started on 2026-10-28 at 17:31. The source was about 13.5 TB, and the stream ran at about 44 MB/s (about 160 GB/h). The script does a preflight (pool ONLINE, ssh works, enough space), takes a fresh source snapshot, runs zfs send … | mbuffer | zfs recv -s (resumable from receive_resume_token), then verifies GUIDs, puts a hold on the base snapshot on both sides and re-points the task.
The ending. The resend finished with no resume token and a hold on its base snapshot. The first scheduled run after it was incremental: from the held 10-28 base to a new 11-02 snapshot, 04:30 to 06:15, about 1 h 45 min for 4 days of changes. GUIDs match on both sides, the hold is in place on both sides, and the offsite pool is healthy. Going forward, the newest common snapshot is held on both sides so retention cannot remove it.
Limit of the encryption. The key file loads automatically at boot. That protects disks that leave the box, not a stolen running box. A recovery copy of the key lives with the DR kit.