Jellyfin and HandBrake: Media Serving, GPU Encoding and a Permissions Puzzle

Purpose

Jellyfin serves media to any device on the network without subscriptions or ads. HandBrake sits alongside it for automated format conversions, using an NVIDIA GPU for fast encoding. Post-processing a newly downloaded movie so it plays on phones and tablets without taxing storage is the same workflow most homelab users hit — download with the arr stack, store on shared storage, transcode on demand.

Jellyfin runs as a Docker container on the GPU host alongside HandBrake. Both read from one ZFS data set on the Proxmox host — Jellyfin via bind mount, the arr stack via NFS. Getting the two sides to coexist uncovered a permissions puzzle at the intersection of NFS root squashing and LXC user namespaces.

flowchart LR
    K3s["k3s cluster<br/>(arr stack)"] -- "NFS mount\nroot_squash → UID 65534" --> NFS["ZFS dataset\n(Proxmox host)"]
    NFS -- "bind mount" --> LXC["unprivileged LXC<br/>(GPU Docker host)"]
    LXC -- "shared mount" --> JF["Jellyfin (docker)"]
    LXC -- "shared mount" --> HB["HandBrake (docker,\nNVENC GPU encode)"]
    JF --> CL["Clients\n(browsers, apps)"]
    HB --> OUT["Converted output\n(480p–2160p)"]

Deployment

Both containers run on the same GPU-enabled Docker host. Jellyfin uses image jellyfin/jellyfin:12.1, running with a healthy status check and an always restart policy. HandBrake runs zocker160/handbrake-nvenc:110x with an NVIDIA GPU device available through the standard nvidia driver passthrough, set to restart unless-stopped. Both log via journald and sit on a shared Docker network for reverse proxy access.

The arr stack lives on k3s in the default namespace, managed by a single Deployment that packages Sonarr (4.0.20), Radarr (v6.4.4.10685), and qBittorrent (5.2.3) alongside Gluetun. This is where the media enters the system before Jellyfin makes it available.

Storage

The server stores the library as one ZFS dataset on the Proxmox host, partitioned into plain folders (Movies, TV, Anime) without per-title pools or separate volumes. This single dataset reaches both sides of the homelab:

• Bind mounted directly into the GPU Docker LXC, so Jellyfin and HandBrake read from the host filesystem with no network hop for local media.
• Exported via NFS to the k3s cluster as a PersistentVolume — the arr apps mount it the way they would any NFS-backed storage.

Jellyfin has read-write access for its cache and metadata directories, while HandBrake mounts several watch folders to trigger automatic transcodes at different resolutions (480p through 2160p). Converted files land in their own output directory.

The Permissions Puzzle

After the initial setup was working, Jellyfin started failing to save subtitles or library metadata in folders the arr apps had just populated: new directories were owned by UID 65534 with permissions 755, so Jellyfin could not write into them. Two system designs collided.

The arr apps in k3s wrote over an NFS mount exported with root_squash. Their containers ran as root inside the pod, but the NFS server mapped every root request to the nobody account (UID 65534). A umask of 022 meant directories were created at 755, files at 644 — owned by nobody and group nogroup, readable by everyone else but writable only by the owner. On its own, that would not have mattered. But Jellyfin runs inside an unprivileged LXC container, which uses Proxmox’s user ID mapping to shift UIDs into the 100000+ range. So Jellyfin was neither UID 65534 nor anyone who inherited write access through a group — it simply could not write.

The fix applied: all arr processes in the Deployment share a ConfigMap that defines their environment variables. Setting UMASK=000 in the shared config means new files are created world-readable and world-writable (777 for directories, 666 for files). The top-level library folders were also opened to mode 777. That is enough — Jellyfin can now write subtitles, metadata, and transcoded thumbnails anywhere. The change has been confirmed in the Sonarr, Radarr, and qBittorrent containers running with umask 0000.

This works, but it is brute force. On a single-user library accessible only over the home LAN, letting anyone write to media folders is not a security concern — there is no attacker on the network. Cleaner alternatives exist: assign a dedicated shared GID mapped into both the LXC user namespace and set as the NFS export group via all_squash with explicit anonuid/anongid. That separates concerns without broad permissions, but requires changes on both the Proxmox side and the NFS export configuration at once. For now, the umask zero covers it.

Notes

• Jellyfin transcodes on playback using its own built-in FFmpeg — no separate worker is needed for streaming. HandBrake fills the other half of the job: permanent format changes on downloaded files before they enter the library proper.
• The GPU device in both containers does double duty: Jellyfin uses it for hardware-assisted transcoding during live playback, and HandBrake relies on NVENC to convert source files into multiple target resolutions without overloading the CPU.
• The umask fix is simple but broad. If the library ever grows beyond one user, the GID-mapping route is worth revisiting.