Homelab
One lesson a day from running a homelab where AI agents do most of the work: what went wrong, what we learned, what went right, and how to do it now.
Each entry is one thing that broke or surprised me, in the same four parts. The “how to do it” part carries the date I last checked it. Follow by RSS .
- Agents in production · FixedThe agent noticed it held the only copyMy git server went down with memory errors. I mentioned it to the agent working on my video project, nothing more. Forty-four seconds later it had written and verified a backup of the commits that existed nowhere else.
- Agent boundaries · Still trueThe agent prepares, I pull the switchWhen my agents hit a permission wall they don't route around it. They write me a short command or a script, I run it, and the last line says: tell Claude to verify and commit.
- Metering & plumbing · Still trueYour logs forget faster than your filesystemReconstructing eight months of my homelab build, the system journal gave me two weeks and agent transcripts about a month. Package logs, file birth times and git went back to February.
- Agents in production · Still trueThe agent imagined a permission error and threw away its workA headless agent drafted a full meeting agenda, reported that it couldn't write the file, and discarded the draft. The denial never happened. The same run had written to that folder ten minutes earlier.
- The box & network · FixedMount ordering stops services and never restarts themPowering on the computer next to it reset my RAID's USB bridge. The disk was back in 18 minutes. The service that depended on it stayed down for three hours, because a clean stop is not a failure.
- Agents in production · Still trueA model alias is a deploy you didn't doOn September 28 my overnight agent got five times faster and stopped poking at repositories it couldn't read. I hadn't changed a line. The model behind the alias had changed.
- Silent failures · FixedThe long-polling bot that short-polled for 36 daysMy Telegram ops bot was meant to hold each request open for 50 seconds. A name collision in my own wrapper made it ask every 3.5 seconds instead, and my DNS sinkhole was the thing that noticed.
- Records & decisions · Still trueRecords are state. Prose is context.Before building a task system I counted how prose had failed me: 14 incidents in two months. The fix wasn't a second surface to update. It was making the record the place where the work lives.
- Agents in production · Still openIt proposed the same one-line fix 15 nights runningMy report-only overnight agent proposed the same one-line fix on 15 of 16 nights. The fix landed 25 days after the bug was filed, in a daytime session. The limit was how fast I decide.
- The agent team · Still trueMy agent saved its memory where I couldn't see itMy editorial agent wrote its memory to Claude Code's machine-local auto-memory instead of the versioned file in the repo. It defaulted to the copy I couldn't review.
- Model pipelines · FixedAsked to score, it asked to rewrite the scorerOn its first scheduled run, a small model asked to score bids wrote a 9 KB regex scorer instead, then asked for permission to install it. Headless, nobody says no.
- The box & network · Still trueSMART said PASSED with 3,504 reallocated sectorsA failed 8 TB drive from my RAID enclosure passed its SMART self-assessment. So did a drive with 75,975 power-on hours. Read the raw attributes, not the verdict.
- Agent boundaries · Fixed"Never rm" was a sentence, and rm workedMy nightly agent's prompt said never mv or rm. I probed it in the same permission mode it runs in: ls, mv and rm all ran. The prohibition was prose.
- Silent failures · FixedFailing four Sundays under a green dashboardA health check exited 1 four Sundays in a row while my status command said nothing needed attention. It was checking schedules, not results.
- Records & decisions · Still trueThe bottleneck was meI assumed my agents were the bottleneck, so I counted: 37 of 51 open tasks were waiting on my decision.
- The agent team · Still trueNine agents planned, five builtMy plan for an AI newsroom had nine agents. For the first weeks the whole team was one session wearing the editor's persona.
- Agent boundaries · FixedIf your agents can write to your backups, so can a bad agentEvery process running as me could write my backup store, including the AI agents the backups were meant to protect against.
- The box & network · FixedTailscale SSH ignores authorized_keysA write-only backup key gave me a shell. With Tailscale SSH on, per-key restrictions only hold on the LAN address.
- Silent failures · FixedOne video took down my LAN's DNS for four daysA transcription job ran out of memory, its retry counter never counted the crash, and DNS ran on the same box.
- Agents in production · FixedA minimum count in a prompt is a quotaMy overnight agent was asked for 3 to 5 ideas and kept landing on exactly 3. A floor in a prompt turns into a target.
- The box & network · FixedThe flaky LAN was my router losing its default routeNew connections failed about once a minute. The router kept its link and lost its route, and only a 30-minute soak showed it.