Bubbles is the Mac Mini that runs our family AI assistant — you text him, he texts back. One morning he stopped texting back. The monitoring said “gateway down,” the AI’s API host looked unreachable, and every signal pointed the same direction: an upstream outage. Not my problem. Wait it out.
Except the API was fine from every other machine in the house. Just not from Bubbles.
The crime scene
Here’s what the actual crime scene looked like. The gateway process was alive and had been for over a day. Pings worked. Inbound SSH worked. But every single outbound connection failed instantly — including connections from Bubbles to itself. A machine that can be reached but cannot reach anything, including its own loopback, is a special kind of broken.
The kernel error told the story: it couldn’t assign a local port. Every TCP connection you open borrows one of roughly 16,384 ephemeral ports, and when a connection closes, the port sits in a waiting state for a few seconds before it’s returned to the pool. I counted the ports stuck in that waiting state: 18,357. Out of 16,384. The pool wasn’t just empty — the janitor that recycles the ports had wedged too, so the count sat frozen, byte-for-byte identical, minute after minute.
Sixteen days of retries
So what was churning through connections fast enough to drain the pool? A process listing sorted by CPU put it in the first row: a one-shot debug query I’d run against Bubbles’ knowledge base — the kind of command that returns in two seconds and you never think about again — had hung at 100% CPU and sat there for sixteen days, failing and retrying, opening and closing connections the entire time. Orphaned, parentless, invisible, and very busy.
The one door still open
The insult on top: there is no way to flush that wedged port table on macOS from userspace. The only fix is a reboot, and I wasn’t home, and the hung state meant normal remote tools were half-useless. The escape hatch was a graceful AppleScript restart over SSH — the one door that was still open. Thirty seconds later Bubbles came back up, ports at 45 instead of 18,357, and started answering texts again like nothing happened.
A reaper, not a bigger buffet
The fix for next time is where it gets fun. The old healthcheck script asked one question — “does the gateway process exist?” — which was true for the entire outage. Useless. The new one runs every sixty seconds and reaps any of these one-shot queries that’s been alive longer than five minutes, or whenever the waiting-port count crosses 8,000. It kills the disease, spares the daemon it protects, and logs the top CPU offender so an unknown runaway still gets caught. I tested it with a decoy hang: reaped in a minute.
I deliberately skipped the “real” sysadmin fix — widening the port range. That just gives a runaway process a bigger buffet. The reaper kills the thing doing the eating.
Total cause of the outage, in the end: one forgotten debug command. Sixteen days. The machine did exactly what I told it to. That’s the whole problem with computers.