Carlos Granier
Back to homelab
Metering & plumbing ¡ Happened

Your logs forget faster than your filesystem

Reconstructing eight months of my homelab build, the system journal gave me two weeks and agent transcripts about a month. Package logs, file birth times and git went back to February.

Still true Accurate as of
Horizontal bar chart of days of history available per source as of October 6: journald 14 days, agent session transcripts 30, syslog and auth.log 30, apt history.log 34, prompt history 176, the kernel ring buffer 178 (only because there has been no reboot), git history 200, ext4 file birth times 235 and dpkg.log 238.

The short version: In early October I tried to reconstruct how this homelab was built, from the first boot in February to now. The system journal reached back 14 days. The agents’ session transcripts reached back about 30. What remembered the year were the things that simply accumulate: dpkg’s log back to February 10, git history, file birth times on ext4, and a kernel ring buffer that had survived only because the box hadn’t rebooted since April.

Get the week's seven in one email, every Monday.

What went wrong

Nothing was misconfigured. journald on Ubuntu keeps a size budget, about 3 GB on this box, and at the rate this box logs that budget is two weeks. syslog and auth.log rotate monthly. The coding agents keep their transcripts for a similar window. Each default is reasonable. Together they mean that the period I most wanted to understand, the first months, had no logs at all.

The sources that did answer were accidents of design. dpkg.log rotates but keeps its archives, so every package install since February 10 was there. File creation times told me when each project directory appeared. git told me what changed and when. dmesg still held the April boot because nothing had restarted the kernel since, and that record is gone now: the box rebooted on October 10.

What we learned

  • Logs are for the last incident, not for history. Rotation is tuned for disk, not for memory.
  • Things that accumulate beat things that rotate. A ledger file that only grows, a records folder in git, package logs with their archives.
  • The ring buffer is a trap. It looks like a log and vanishes at the first reboot.

What went right

The reconstruction worked with the sources I had, and it produced a dated timeline of the build from February to October. It also changed how the homelab records itself. State lives in task records committed to git. The release job and the newsletter tool write one line per run to ledger files that never rotate. The question of what happened in March now has somewhere to go, which it didn’t when I asked it in October.

How to do it (as of October 2026)

  1. Decide what history you’ll want in a year, before you need it: when things were installed, when services started and stopped, what the agents did.
  2. Give journald an explicit retention: SystemMaxUse= for disk and MaxRetentionSec= for time, so the window is a decision rather than a side effect.
  3. For anything you’ll want to reconstruct, write one line per run to a file that only grows, outside log rotation.
  4. Put state in git: records, decisions, config. A commit history is the cheapest long-term log you’ll ever get.
  5. If dmesg holds something you care about, copy it out today. One reboot and it’s gone.

Status

Still true. The journal on this box still runs about two weeks. The records, the ledgers and git carry the rest.

The week's seven, by email

Entries post here daily. Every Monday, one email with the past week's seven. Unsubscribe anytime.

Let's Connect