Carlos Granier
Back to homelab
The box & network · Happened

The flaky LAN was my router losing its default route

New connections failed about once a minute. The router kept its link and lost its route, and only a 30-minute soak showed it.

Fixed Accurate as of
Bar chart of refused connections per minute on two machines. About 3 a minute at baseline, a misleading 0 to 0.4 after a cable swap in short runs, still about 1 a minute in a 30-minute soak, and zero in two 30-minute soaks after the ISP gateway went into bridge mode.

The short version: my mesh Wi-Fi router, sitting behind the ISP’s gateway, briefly lost its default route about once a minute and answered new connections with “network unreachable”. Bridge mode on the ISP gateway fixed it. Short tests said it was fixed before it was.

What went wrong

My dev proxies kept failing with connect ENETUNREACH for a second or two at a time, then recovering. They showed up as 502s in testing, and one git push failed and then worked on retry. I suspected Tailscale’s routing first, then the kernel.

Neither was it. A UDP probe that opened a new socket for every packet and read the kernel’s error queue (IP_RECVERR) named the sender: the Wi-Fi router itself was answering new outbound flows with ICMP “net unreachable”. It did that 8 times in 1,496 flows over 150 seconds, in pairs within the same millisecond across unrelated destinations. A second machine on the LAN saw the same thing.

About 0.3% of new connections failed. Established connections barely noticed, which is why only a proxy that opens a fresh upstream connection for every request ever surfaced it.

What we learned

  • Timing tells you who sent an error. A rejection from my own kernel comes back in about 0.06 ms. These came back in 0.7 to 1.8 ms, one round trip to the router. An error that needs a round trip to arrive was sent by something one hop away.
  • Short tests under-sample a once-a-minute fault. After a cable swap, two 160-second runs showed 1 failure and then 0. It looked fixed. A 30-minute soak still showed about one failure a minute.
  • Probe the next hop alongside the internet. I added the ISP gateway as a probe target and rotated the order every round. In 3,835 rounds the gateway failed 0 times while internet targets failed 15. The router kept its link and lost its route. The gateway’s own event log was empty.

What went right

I measured before changing anything, so each router-side step (restart, cable swap, bridge mode) had a before and after number. And after bridge mode gave the LAN public IPv6 for the first time, I tested inbound connections from a phone on cellular. The router’s firewall blocks them.

How to do it (as of October 2026)

  1. Time the failure. A local reject is near-instant; a reply from a router takes a round trip.
  2. Find the sender: a UDP socket with IP_RECVERR reports which address sent the ICMP error.
  3. Probe the next hop and a few internet targets together, rotating the order each round, so a lost route and a lost link look different.
  4. Compare 30-minute soaks with 30-minute soaks. Two short clean runs prove little against a fault that fires once a minute.
  5. If you remove a double NAT, test inbound from outside your network afterwards. A topology fix can change what’s exposed.

Status

Fixed on October 5, 2026: the ISP gateway is in bridge mode and the double NAT is gone. A confirmation soak the next day ran 11,250 rounds on one machine and 11,223 on the other with zero failures.

Let's Connect