The agent noticed it held the only copy

The short version: On October 15 the git server I push everything to started refusing pushes, then disappeared from the network entirely. I told the agent working on my video project that it would be unreachable for a while, and nothing else. It worked out that ten commits now existed only on the machine it was running on, wrote a git bundle of both branches to the RAID, verified it, and reported back. Forty-four seconds, unasked.

Get the week's seven in one email, every Monday.

What went wrong

The first symptom came at 18:00: a push failed with remote unpack failed: unable to create temporary object directory. SSH worked; the server couldn’t write. By 19:11 it was gone. Port 22 timed out, and so did a Tailscale ping. The cause turned out to be RAM on the host: its filesystem had caught write-time corruption at 17:25 and forced itself read-only. A week of memory tests found one failing module, and the host now runs on the other.

Meanwhile every project on this box was pushed up to date except one. The video project had been committing all afternoon and was ten commits ahead of a remote that no longer existed. Its newest work was also its least backed-up work, because the thing that backs it up is the remote.

What we learned

• When the remote is down, your newest work is your least backed-up work. The exposure is exactly the commits since the last successful push.
• Read globally, act locally. At 18:03 the agent filed the unpack error on the infrastructure board instead of poking at the git server, which wasn’t its project. At 19:13 it protected its own work, which was.
• Improvising a copy is fine. Improvising into the backups isn’t. It wrote the bundle to its own project folder on the RAID. The backup store is root-owned, and no agent can write there. It didn’t try.

What went right

The bundle held main and the 3D branch, 154 MB, created and verified at 19:13:38, 44 seconds after my message. A read-only exposure check that evening confirmed the video project was the only repository with commits that existed on one machine; ten other repositories were zero ahead. The agent refreshed the bundle the next day as it kept committing. On October 17 the server came back on its remaining memory module and the push went through cleanly.

How to do it (as of October 2026)

1. When a remote goes down, list what is ahead of it: git log origin/main..main --oneline in every repository that pushes there.
2. For anything ahead, git bundle create <file> --all to a different disk, then git bundle verify <file>. A bundle is one file and needs no server.
3. Give agents a place where they may write backups, and keep the real backup store out of their reach. Both halves matter.
4. Tell your agents what’s happening around them, even in one line. This one acted on a sentence.
5. Retire the bundle when the push succeeds, and say so in the record.

Status

Fixed. The server is back, the commits are pushed, and the host’s memory fault has its own open record.