Jellyfin and HandBrake: Media Serving, GPU Encoding and a Permissions Puzzle
Purpose
Jellyfin serves media to any device on the network without subscriptions or ads. HandBrake handles automated format conversions on a separate Docker host, using an NVIDIA GPU (NVENC) for fast encoding. Post-processing a newly downloaded movie so it plays on phones and tablets without taxing storage is the same workflow most homelab users hit — download with the arr stack, store on shared storage, transcode on demand.
Jellyfin runs as a Docker container on the iGPU Docker host, and HandBrake runs on a separate Docker host that kept the NVIDIA card. The two used to share one GPU host; that host was retired in November 2026 (brief). Jellyfin reads the library through bind mounts into its LXC, and the arr stack writes to it from k3s over NFS. Getting the two sides to coexist uncovered a permissions puzzle at the intersection of NFS root squashing and LXC user namespaces.
flowchart LR
K3s@{ img: "/assets/img/icons/kubernetes.svg", label: "k3s cluster<br/>(arr stack)", pos: "b", w: 40, h: 40, constraint: "on" }
NFS@{ img: "/assets/img/icons/proxmox-light.svg", label: "Media library<br/>(NFS share)", pos: "b", w: 40, h: 40, constraint: "on" }
LXC@{ img: "/assets/img/icons/docker.svg", label: "unprivileged LXC<br/>(iGPU Docker host)", pos: "b", w: 40, h: 40, constraint: "on" }
JF@{ img: "/assets/img/icons/jellyfin.svg", label: "Jellyfin (docker,<br/>Quick Sync)", pos: "b", w: 40, h: 40, constraint: "on" }
HB@{ img: "/assets/img/icons/handbrake.svg", label: "HandBrake (docker,<br/>NVENC, RTX 4060)", pos: "b", w: 40, h: 40, constraint: "on" }
K3s -- "NFS mount\nroot_squash → UID 65534" --> NFS
NFS -- "bind mount" --> LXC
LXC -- "shared mount" --> JF
HBH["Separate Docker host<br/>(RTX 4060)"] --> HB
GT["handbrake-guard.timer<br/>(TrueNAS shares mounted?)"] --> HB
JF --> CL["Clients\n(browsers, apps)"]
HB --> OUT["Converted output\n(480p–2160p)"]
Deployment
The two containers now run on different Docker hosts. Jellyfin runs on the iGPU Docker host, an LXC on the second Proxmox node with an Intel Iris Xe, using image jellyfin/jellyfin:12.4, with a healthy status check and an always restart policy. HandBrake runs on a separate Docker host that kept the RTX 4060, image zocker160/handbrake-nvenc:110x with the nvidia driver passthrough. It starts only when its TrueNAS shares are mounted, which a handbrake-guard.timer checks (brief). On the brief’s jobs the H.265 NVENC 1080p preset ran at roughly 300–380 fps.
Jellyfin on Quick Sync
Jellyfin’s hardware transcoding runs on Quick Sync now. The brief’s test transcode was a 4K Dolby Vision file to 720p H.264: VA-API decode, tonemap_vaapi, and h264_qsv in low-power mode, at 476 fps (19.9x realtime) (brief).
The arr stack lives on k3s in the default namespace, with Sonarr, Radarr and qBittorrent behind Gluetun. This is where the media enters the system before Jellyfin makes it available.
Storage
Jellyfin mounts several media directories (Anime, Anime2, AnimeMovies, Films, TV) from the Docker host, plus its own cache and transcode directories. The library reaches both sides of the homelab:
- Bind mounted into the iGPU Docker LXC, so Jellyfin reads from the host filesystem with no network hop for local media.
- NFS-mounted by the arr apps in k3s, which is where the root-squash story below comes from.
- HandBrake runs on its own host with its TrueNAS shares mounted there. The guard timer holds it back until they are present.
Jellyfin has read-write access for its cache and metadata directories, while HandBrake mounts several watch folders on its own host to trigger automatic transcodes at different resolutions (480p through 2160p). Converted files land in their own output directory.
The Permissions Puzzle
After the initial setup was working, Jellyfin started failing to save subtitles or library metadata in folders the arr apps had just populated: new directories were owned by UID 65534 with permissions 755, so Jellyfin could not write into them. Two system designs collided.
The arr apps in k3s wrote over an NFS mount exported with root_squash. Their containers ran as root inside the pod, but the NFS server mapped every root request to the nobody account (UID 65534). A umask of 022 meant directories were created at 755, files at 644 — owned by nobody and group nogroup, readable by everyone else but writable only by the owner. On its own, that would not have mattered. But Jellyfin runs inside an unprivileged LXC container, which uses Proxmox’s user ID mapping to shift UIDs into the 100000+ range. So Jellyfin was neither UID 65534 nor anyone who inherited write access through a group — it simply could not write.
The fix applied: all arr processes in the Deployment share a ConfigMap that defines their environment variables. Setting UMASK=000 in the shared config means new files are created world-readable and world-writable (777 for directories, 666 for files). The top-level library folders were also opened to mode 777. That is enough — Jellyfin can now write subtitles, metadata, and transcoded thumbnails anywhere. The change has been confirmed in the Sonarr, Radarr, and qBittorrent containers running with umask 0000.
This works, but it is brute force. On a single-user library accessible only over the home LAN, letting anyone write to media folders is not a security concern — there is no attacker on the network. Cleaner alternatives exist: assign a dedicated shared GID mapped into both the LXC user namespace and set as the NFS export group via all_squash with explicit anonuid/anongid. That separates concerns without broad permissions, but requires changes on both the Proxmox side and the NFS export configuration at once. For now, the umask zero covers it.
Notes
- Jellyfin transcodes on playback using its own built-in FFmpeg — no separate worker is needed for streaming. HandBrake fills the other half of the job: permanent format changes on downloaded files before they enter the library proper.
- The GPUs no longer share duty. Jellyfin uses the iGPU’s Quick Sync for live playback transcoding, and HandBrake keeps the RTX 4060 and NVENC.
- If you are weighing an iGPU against a discrete GPU: Quick Sync covers a Jellyfin library well, while batch encoding is where the dedicated card still earns its place. See also Frigate and Immich.
- The umask fix is simple but broad. If the library ever grows beyond one user, the GID-mapping route is worth revisiting.