Bubbles — the Mac Mini that runs our family AI — keeps a knowledge base: a few hundred pages of notes, embedded and indexed, so the assistant can actually know things about our life instead of guessing. On May 26th that database corrupted and I rebuilt it from scratch.
It survived twenty-eight hours.
Killed at 10:03:49
At 10:03:49 the next morning, macOS’s launchd sent a termination signal to the gateway process. The gateway died, and it took its child — the database server, mid-checkpoint — down with it. I know the exact second because the database directory told me: the transaction log, the control file, and about eighty data files all share the same modification timestamp, 10:03:49. Writes that need to be atomic for recovery to work got sliced in half within a single second. After that, every attempt to open the database died with the same useless WASM-runtime error, which is the software equivalent of a shrug.
The dark comedy is how I found it. I wasn’t investigating the database at all. I was investigating why two processes were each eating a gigabyte of RAM at 99% CPU — and those turned out to be zombie queries from four days earlier, hung by a bug that upstream had already fixed, still faithfully holding handles on the previous corrupted database. I went in to kill two zombies and found a fresh corpse.
My own watchdog did it
And the killer? Almost certainly my own healthcheck agent — the watchdog whose job is to restart the gateway “if dead.” Something tripped its threshold, it fired a kill signal at a process that was fine, and the kill landed on a database in the middle of writing to disk. My safety system caused the outage. I’d like to say that’s rare, but the entire genre of postmortem writing suggests otherwise.
Twenty-five minutes to rebuild
Recovery took about 25 minutes, most of it re-importing and re-embedding 560 pages of notes. Along the way I upgraded the tool to the version with the zombie-query fix, re-pinned the one dependency version that actually works on this macOS release (the newer install helpfully reverts it to the broken one — a trap I now step over on autopilot), and verified everything: 597 pages, 2,753 chunks, 100% embedded.
Nothing is fixed
Here’s the honest ending: nothing is fixed. The upstream project patched the hang bug, but graceful shutdown on a kill signal mid-write — the thing that actually destroyed two databases in two days — is still an open issue. The architecture still runs the database as a child of a process that other software is allowed to kill. My mitigation options are all written down (decouple the lifecycles, tame the trigger-happy watchdog, sync less often so the write window shrinks), and I’ve implemented exactly none of them yet.
So the current state of the art is: I have a fast, well-rehearsed procedure for rebuilding the brain, and a brain that will need it again. Some problems you solve. Some you just get quick at.