Write-ups · Root cause · 7 Sept 2026
Docker containers silently lose the GPU after a systemd reload
Every GPU container on my server lost its device access at the same moment and nothing logged an error. The trigger was an unrelated systemctl daemon-reload.
- What they said
- Three GPU services died at the same minute and nothing logged an error.
- What it was
- A systemd reload wiped device rules Docker never knew about.
Symptom
Video transcoding on my media server started failing with FFmpeg exited with code 187. The transcode log had the real message:
cu->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
On the host, nvidia-smi showed both cards healthy. Inside the container it said Failed to initialize NVML: Unknown Error. The device nodes were still there. The photo library's machine-learning container and my LLM container had the same problem, and all three had been fine an hour earlier.
What changed
Nothing about the containers. The only event in the journal at the right time was me installing a systemd timer for something unrelated, which runs systemctl daemon-reload.
Cause
Docker on this machine uses the systemd cgroup driver on cgroup v2. When a container asks for a GPU, the NVIDIA container runtime adds the device permissions with a hook, behind systemd's back. systemd does not know about those rules. On a reload it re-applies the device policy it does know about for every scope, and the GPU permissions are gone. The process keeps its open file handles, so it fails only on the next CUDA initialisation. That is why it looks random.
Fix
Tell Docker about the devices explicitly, so Docker registers them with systemd and they survive a reload:
services:
jellyfin:
devices:
- /dev/nvidia0:/dev/nvidia0
- /dev/nvidia1:/dev/nvidia1
- /dev/nvidiactl:/dev/nvidiactl
- /dev/nvidia-uvm:/dev/nvidia-uvm
- /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
- /dev/nvidia-modeset:/dev/nvidia-modeset
Recreate the container, then prove it:
docker exec jellyfin nvidia-smi -L # lists the GPUs
sudo systemctl daemon-reload
docker exec jellyfin nvidia-smi -L # still lists the GPUs
What I took from it
The failing component and the cause were in different layers, and the only link between them was a timestamp. I now treat "what else happened at that minute" as the first question, and I test a fix by repeating the trigger, not by checking that the symptom went away.