One video took down my LAN's DNS for four days

The short version: a transcription job tried to decode a 6-hour video in one pass, hit about 14 GB on a 15 GB server and was killed by the kernel. Its retry counter only counted finished runs, so the kill never counted and it tried again every 20 minutes for four days. My LAN’s DNS ran on the same box.

Get the week's seven in one email, every Monday.

What went wrong

A job on my home server transcribes audio and video with Whisper. A timer starts it every 20 minutes, and it gives up on a file after a few failed attempts.

I dropped in a 6-hour video. Whisper decoded it in one pass, memory climbed to about 14 GB on a machine with 15 GB usable, and the kernel’s out-of-memory killer ended it with SIGKILL. The attempt counter was written after the run finished. A process killed mid-run never gets there, so as far as the job knew, it had never tried. It tried again 20 minutes later, and again, for four days.

Swap filled up and stayed full. Pi-hole, the DNS server for my whole LAN, runs on the same machine, so the house’s DNS went down with it. Nothing was watching for failed system services, so nobody noticed.

What we learned

• Bookkeeping that runs after success doesn’t protect you from a crash. Count the attempt before the work starts.
• A memory cap turns “kill the box” into “kill the service”. Without one, a single runaway job takes everything on the machine with it.
• Chunk the input before buying RAM. The job didn’t need 14 GB; it needed smaller pieces.
• DNS on a workload box makes every memory event a network outage.

What went right

All four fixes landed the same evening, and the same 6-hour file then went through cleanly: 18 chunks, 83 minutes, 63,932 words, all inside a 4 GB cap that the old path had blown through in 60 seconds.

How to do it (as of October 2026)

1. Increment the attempt counter before running the job, not after.
2. Cap heavy batch services with systemd: MemoryMax=4G, MemoryHigh=3G, MemorySwapMax=256M.
3. Split long media with ffmpeg (20-minute segments here), transcribe each piece, then join the text.
4. Watch how each service’s last run ended, not only whether it’s scheduled. A oneshot that failed should show as failed somewhere you look.
5. If you can, keep DNS off the machine that runs heavy jobs. If you can’t, cap everything else on it.

Status

Fixed on July 30, 2026. DNS still runs on the same machine; the memory caps are what keep one job from taking it down again.