Skip to content

NAS reboot & NFS (nfsd) hang recovery

TerraMaster F4-423 (192.168.0.237), NFS server for VM 208's media stack.

Symptom: alert flood, media containers down, NAS high load

A flood of media_container_missing on 208 while 208 itself is healthy = the NAS's nfsd is hung in D-state (uninterruptible sleep). NFS stops accepting connections → NFS-dependent containers on 208 crash in cascade. No data risk (the RAID is fine).

Diagnose (on the NAS):

ps -eo stat,comm | grep nfsd | grep -c D   # >0 = hung
ss -tln | grep 2049                        # LISTEN with high Recv-Q = wedged

Healthy = nfsd_D=0, Recv-Q 0. Metrics signature: load1 pinned at ~10 (the D-state threads inflate load) while disk I/O is flatlined = a kernel-level I/O stall, not overload.

Cure: reboot the NAS (only option)

A D-state hang can't be killed (not even kill -9). The only cure is a NAS reboot.

GOTCHA — SSH cannot reboot it. The "root" reachable over SSH on TerraMaster is uid 9999 (a sandbox). reboot / systemctl reboot → polkit "Interactive authentication required"; reboot -f → "Need to be root"; sysrq / nfsd thread bounce → "Permission denied". The reboot must go through the TOS web UI (its backend runs as real uid 0) — pick Restart, not Shutdown (a shutdown can leave it powered off).

Remote reboot without the panel: nas-reboot.sh

Lives in ~/scripts/nas-reboot/ on VM 208. TOS 6's login is a bunker (RSA with a time-mangled public key + CSRF + X-Security-Code), so instead of re-implementing that crypto the script drives a headless Chromium through the real login page, then reuses the session cookie for PUT /v2/system/reboot.

cd ~/scripts/nas-reboot
./nas-reboot.sh            # DRY RUN — verify login + authenticated call, NO reboot
./nas-reboot.sh --reboot   # actually reboot the NAS

Credentials in nas-reboot.env (chmod 600). Node + puppeteer (Chromium) vendored in the same dir. Robust to TOS crypto changes because it runs the panel's own JS.

After the NAS comes back

nfs-health-monitor.{sh,service,timer} on 208 auto-recovers the stack. Two gotchas:

  1. It gives up at attempt 30 (~12–13 min). If the NAS is slower than that, restart the crashed containers by hand: docker start $(docker ps -aq -f status=exited).
  2. Stale bind mounts (fixed 2026-07-16). When the NFS remounts, containers that stayed "Up" keep their bind pointing at the pre-remount (empty) dir → Sonarr reports "Importing after download will exceed available disk space" even though the NAS has TB free (docker exec sonarr ls /data is empty while host /mnt/nfs_media is full). The monitor now runs rebind_nfs_containers() on recovery — docker restart on every container binding /mnt/nfs_media (discovered dynamically). Manual trigger: ~/scripts/nfs-health-monitor.sh rebind.

Verify recovery: alertmanager container up + Prometheus firing=0.

Long-term hardening (not yet done)

  • Phase 2: wire nfs-health-monitor to auto-trigger nas-reboot.sh --reboot after N min of confirmed D-state hang (rate-limited, Telegram notify) — start in "detect + alert" mode before promoting to auto.
  • Smart plug as a last-resort power-cycle for when even a graceful reboot wedges (BTRFS is CoW and survives a hard cut).