On July 16th, a routine image update quietly killed the Cloudflare tunnel on Randy. I found out on July 19th. Not from an alert. From the family shopping list not loading.
Three days. Four public services — the shopping list, media requests, analytics, and remote Plex — all dark, and the house said nothing.
A network that no longer existed
Here’s the failure, because it’s a good one. The tunnel container had been created back when my media stack had a different name, and its network attachment — 1_media-net — was pinned at creation time. At some point I’d renamed the stack, the old network went away, and the running container didn’t care; it was already attached. It ran happily for weeks on a network that technically no longer existed, like a guy still using a gym membership from a location that closed.
Then the image updated. The update recreated the container, the new container tried to join 1_media-net by name, and there was no such network anymore. So it just… didn’t start.
Created, never started
And this was the kicker: it failed with zero logs. The container sat in Docker’s “Created” state — not running, not restarting, not crashing, just created, the way a form is created before anyone fills it out. Nothing wrote a line anywhere. The clue, when I finally went looking, was a timestamp: the container’s StartedAt was zero. It had never started. There’s no alert for a thing that never began.
My monitoring made it worse in an instructive way: of the four services riding that tunnel, I was only monitoring one — and I was monitoring it in a way that didn’t catch this. So one blind spot hid three others behind it. Silent failures travel in groups.
The insultingly small fix
The fix was almost insultingly small. Delete the container, recreate it on Docker’s default bridge network — which always exists, and will survive any future fit of stack-renaming I indulge in — with the same mounts and restart policy. Total repair time, a few minutes. Total diagnosis time, most of a morning. The usual ratio.
Then the actual work: a proper uptime check on the tunnel itself, polling every five minutes, wired to notifications that reach me. The monitor expects the specific response a healthy-but-locked-down endpoint returns, so it catches “tunnel dead” without me having to expose anything to make it checkable.
Presence of success
The lesson I keep re-learning, in new costumes each time: the failures that hurt aren’t the loud ones. A crashing container restarts itself, or at least screams into a log. A container in “Created” state is a tree falling in a forest with no microphones. Every system I run has some version of this — a thing that can stop without ever technically failing — and the only defense is monitoring that checks for presence of success rather than absence of errors.
Anyway. The shopping list works again. Nobody in the house noticed it was gone except me, which either means the system isn’t that important or the family has learned to route around my infrastructure. I’ve decided not to investigate which.