I can text Bubbles — our family AI — “where are the kids with their summer homework?” and it answers from the actual schedule, because I gave it the actual schedule. I can ask what we decided about a business question three months ago and it quotes the note where I decided it. Who someone is, what a project’s constraints were, what I paid for a thing in 2024 — it knows, because I handed it everything: my entire notes vault, 596 pages and 2,934 chunks of projects, decisions, people, and write-ups going back years, re-synced every fifteen minutes.

An assistant that can’t remember your life is a stranger every morning. This one remembers. Honestly, it’s my favorite thing in the whole setup — the moment the answers started coming back from my own notes, the thing stopped feeling like a chatbot and started feeling like a second brain with my handwriting in it.

Getting there took the setup plus two fixes, and neither fix was the one I expected. Both are worth sharing, because they’re the difference between “technically has access to everything” and “actually uses it.”

Why a brain and not just a bigger context window

Two reasonable objections up front.

Why not paste everything into the prompt? Because 596 pages of notes doesn’t fit, and even when a chunk of it does, stuffing a model’s context with mostly-irrelevant text measurably degrades what it does with the relevant part. Retrieval isn’t a workaround for small context windows. It’s a way of deciding what deserves attention.

Why not a plain vector database? That was my first version — a small custom package doing semantic search over the vault. It worked. What I moved to is GBrain, Garry Tan’s open-source personal knowledge brain, self-hosted on my own hardware. It adds three things on top: a self-wiring knowledge graph where every page write extracts entity references and creates typed links (works_at, founded, advises, attended) with zero LLM calls; typed claims where facts written in a fenced block become first-class columns, so “what was that number back in March” is a query instead of a rummage; and hybrid retrieval with a reranker, which is the part that actually determines whether any of this is useful.

The retrieval stack, concretely: embeddings at 1,280 dimensions, plus a keyword index, combined with reciprocal rank fusion, plus multi-query expansion driven by a small fast model, plus a dedicated reranker model over the top 30 candidates. The provider’s own benchmarks put that at 97.9% recall in the top 5, roughly 31 points better than embedding-only.

The default install was set to a cheaper mode that skips the reranker. I turned it on immediately, because the reranker reshuffles about 60% of top-1 results — it is the quality. The whole indexing pass cost about two cents. Each query costs three hundredths of a penny. This is not where the money goes.

Fix one: the corpus was poisoning itself

Months in, search quietly went to garbage — but only for short queries.

I asked what book my kid was reading over the summer. The brain returned journal one-liners. “Beer cover.” “hamburger help me.” Actual entries, verbatim, ranked above the page literally titled with the summer reading list.

Everything downstream was healthy. Embeddings fine — a nearest-neighbor test from a fresh chunk returned perfect neighbors. Keyword index current. Reranker working. Sync loop running. All the parts passed, and the whole thing was broken.

The culprit was about 87 pages of under sixty words each — old journal fragments and comedy scraps, one line apiece. Very short text embeds near the centroid of short text generally, which means those pages scored 0.86 to 0.97 against essentially any short query. Not a bug in the math. That’s what the math does. They flooded the entire top-10, and while the reranker could sometimes rescue a query, the query-expansion step generated enough candidates that the true page never survived long enough to reach it.

The most instructive part: the “cheap” mode — raw vectors, no reranker — was the worst of all. The exact configuration you’d pick to save three hundredths of a penny is the one with no defense against this.

The fix was corpus surgery, not code. 55 journal one-liners were consolidated into six per-year rollup pages. 18 comedy fragments became one page. A dozen empty stubs were deleted. Content preserved, originals in git history. Both benchmark queries went back to returning the correct page at #1.

Two things fell out of that cleanup that I’d hand to anyone building this:

Tiny chunks are as poisonous as tiny pages. I’d imported my per-project instruction files as a second source — good curated context, obviously useful. They chunk into tiny code-fence fragments, which recreates the exact same centroid pathology one level down. Pulled back out the same day. The content is fine; the shape is wrong, and I don’t have a chunking strategy for it yet.

Embeddings drift silently. The audit found 32 chunks that had never been embedded at all — nothing flagged them, they were simply absent from search forever. The sync job now runs an explicit “embed anything stale” pass every single cycle. And the query cache happily serves pre-surgery garbage after a corpus change, so clearing it is now part of the procedure.

There’s now a weekly lint that fails if new tiny pages start accumulating. The corpus is an input you have to maintain, not a pile you dump things in.

Fix two: making sure something asks

Retrieval was fixed. The assistant still didn’t use it.

I asked where the kids were with their summer homework. It answered from tasks, reminders, notes, the calendar — everywhere except the brain, which had a dedicated page for each kid.

Here’s the actual cause, and it’s a structural problem anyone wiring an MCP-style tool server into an agent will hit. The brain exposes 102 tools. The agent runtime defers large tool sets behind a search step rather than loading every schema into context — sensible, necessary at that scale. But it means the model only discovers the brain’s tools when the message contains something matching them. In practice: when I literally typed the word for the brain by name.

Nobody phrases a real question that way. “Where are the kids with their homework” contains no keyword that surfaces a memory tool, so the model never learned the tool existed on that turn, so it answered from whatever was already in front of it. I’d written a directive in the assistant’s always-loaded profile telling it to check the brain first. Directives lose to structure. It cannot call a tool it hasn’t been shown.

The fix is about thirty lines: a plugin registering a pre_llm_call hook. Before every turn, for any message over sixteen characters, it runs a query against the brain and injects the top three pages scoring above 0.72 directly into the turn as context.

The details that make it safe to leave running:

  • Expansion off (--no-expand) — this is a reflex, not a research task, and it needs to be cheap
  • ~2 seconds typical, hard 8-second timeout
  • Fail-open: if the brain is down, slow, or angry, the turn proceeds with no injected context rather than erroring
  • Installed as a user plugin, not a fork of the runtime, so it survives updates

The model stopped needing to decide to consult its memory. Now the memory arrives with the question, the way yours does.

I tried the built-in feature for this first. It returned nothing on my corpus — it works off entity aliases and my pages have no alias metadata. Worth ten minutes to check before writing your own; also worth not stopping there when the built-in doesn’t fit.

The part that’s still fragile

The brain corrupted itself twice inside 24 hours in its first week — a process killed mid-checkpoint left the database unopenable, twice, and that story is its own post. Add to that a pinned database dependency that a routine upgrade would silently revert, requiring a manual re-pin after every update.

That was all a consequence of running an embedded, file-based database with a single-writer lock held by a long-lived process on a Mac Mini. It doesn’t run that way now — the brain lives in an actual Postgres container on the home server, which is a boring answer that removes an entire category of failure.

The thing I’d tell my past self: an embedded database is a wonderful choice for something you can afford to rebuild and a bad one for something you’re treating as memory. The brain was always rebuildable from the vault, which is the only reason those incidents were inconvenient instead of catastrophic. The vault is the source of truth; the brain is a derived index. Keep it that way and you can lose the brain on a Tuesday and shrug.

What it turned into

The best sign it was worth building is that other things started reading it.

A search app I built later queries the brain’s database directly — a dedicated read-only role, SELECT on just the pages and chunks tables, row-level security policies pinning it there — to search my notes and photos in the same box as tasks and lists. It runs a synthetic health check every five minutes that fires a known query and asserts a known page comes back, specifically to catch schema drift in the brain loudly instead of silently.

Which is the arc, really. It started as memory for one assistant and became a substrate — the place my notes are searchable from, by anything I write next. That only happened because it stopped being an embedded file on a laptop and became a database with a boring uptime story.

The retrieval fix made it correct. The reflex made it useful. Everything since has been other programs finding out it’s there.