The short version: On September 5 I turned on the machine next to my homelabâs RAID enclosure and the enclosureâs USB bridge reset. ext4 aborted, systemd unmounted the array, and 18 minutes later it remounted on its own. The capture webhook that depends on that mount stayed dead for about three hours. systemd had stopped it cleanly when the mount vanished, and nothing told it to start again.
Get the week's seven in one email, every Monday.
What went wrong
The webhookâs unit has RequiresMountsFor= on the RAID path, which is correct: donât run if the disk isnât there. When the mount disappeared, systemd stopped the service, and it stopped cleanly. Restart=on-failure looks for failures, so it never fired. When the mount came back nothing referenced the service, so nothing started it.
Immichâs Postgres, running in a container with a bind mount into the same array, had a worse evening. It kept its file handles into a filesystem that had gone away and threw I/O errors until I restarted it by hand, after which WAL recovery came back clean.
A third drop, on October 9, was mechanical rather than electrical: I unplugged a display cable next to the enclosure and the USB cableâs seating gave. That time Dockerâs restart policy restarted the database container while the array was still unmounted, and Docker recreated the missing bind-mount source on the root SSD. Postgres found an empty directory and initialised a fresh cluster there, 47 MB of nothing. The real database on the RAID recovered by WAL replay once I stopped the stack and moved the stray cluster aside.
What we learned
- A dependency says when to stop, not when to start.
RequiresMountsFor=is half a relationship. The mount unit needs aWants=pointing back at the service, or the service only ever goes one way. - Containers donât notice remounts. A bind mount is resolved when the container starts. If the host path disappears and returns, the process inside is still holding the old inode.
- Docker fills gaps you didnât want filled. With the short bind syntax it creates a missing host path for you. On a server where that path is supposed to be a mounted disk, this is a trap.
What went right
Both fixes went in on October 9 and 10. A drop-in on the mount unit, Wants=capture-webhook.service, so a remount starts the webhook. The three RAID binds in the compose file switched to long syntax with create_host_path: false, so a missing disk is an error instead of a fresh directory. And on October 10 the enclosureâs USB cable was replaced and moved to another port, with a clean fsck at boot, and the array has stayed mounted since.
How to do it (as of October 2026)
- For every service that
RequiresMountsFor=an external disk, add a drop-in on the mount unit withWants=<service>under[Unit]. - Make the mountâs absence loud:
create_host_path: falseon every bind into it, and a marker file inside the mount that your services check. - After any unmount event, restart the containers that bind into it, even if they look alive.
- Treat USB enclosures as fragile: strain relief, their own power strip, and a port you wonât brush against.
- Keep an exposure note: which services die with this disk, and what starts them again.
Status
Fixed, pending proof. The drop-in and the compose change are installed, and there has been no drop since October 10, so the restart path hasnât been exercised for real yet.