Every "AI memory" product you've seen is, underneath the branding, a bigger notepad. It remembers what was said — your preferences, your past chats, some facts — and hands them back so the model doesn't ask twice. Useful. Genuinely.
But a notepad has a ceiling, and we're about to hit it hard. As agents do more autonomous work the bottleneck stops being recall of facts and becomes the one thing a notepad structurally cannot do: continuity of judgment.
A memory that remembers what was said lets an agent repeat itself. A memory that remembers what was decided, why, and how it turned out — and can refuse to act against those decisions — lets an agent, or a team, get better over time instead of relearning the same lessons every Tuesday. That's the bet I'm making with Rekall: not a bigger notepad for AI, but a governed decision substrate that gives autonomous systems memory and judgment.
Underneath that bet is a deeper one I care about more: memory should be infrastructure, not a feature of one tool. A universal layer that travels with you across providers, teams and agents. One engine, three doors — or really three memory scopes: personal (never repeat yourself across your own AI tools), hive (the shared collective any AI on the team can reach — the hive mind), and agent (a persistent substrate for agents to coordinate on). (Hold that "three" — I later found a fourth I didn't see coming.)
The founding bet: five layers, five databases, one mind
The first real call I made set the tone for everything after it: don't build a vector DB with a UI. I wanted to model AI memory the way cognitive science models human memory — distinct systems for distinct jobs — and pick the storage tech whose physics fit each job.
- Short-term — Redis (ephemeral, ~3h TTL) · working memory · recent session context — fast, disposable
- Long-term — Qdrant vectors · declarative memory · facts/preferences, semantic search
- Episodic — Postgres (full-text + temporal) · autobiographical memory · what happened, and when
- Semantic — Neo4j / Graphiti graph · conceptual knowledge · entities + relationships, traversal
- Procedural — Postgres · skills & habits · learned workflows — the "how-to"
Each layer has different physics. Short-term wants speed and expiry. Long-term wants similarity. Episodic wants time and full-text. Semantic wants traversal. Procedural wants step-sequences. One database can't be excellent at all five — so I built five, and let each do the one thing it's actually good at.
An honest note, because I'd rather tell the real story than the heroic one. Rekall didn't begin life as "a product." It began as me poking at an idea I wasn't sure was an idea — the kind of experiment that usually dies in a folder called scratch and never gets mentioned again. By the time I started committing to this repo in earnest, the five-layer engine was already standing, because I'd already built it in the scrappy, un-recorded mucking-about that every real project has and no git log ever shows. The first clean commit lands all five memory types, an MCP server and a dashboard in one go — not because anyone handed me a foundation, but because the experiment had crossed the line from "might be a thing" to "oh, this is a thing," and I finally started treating it like one. The interesting months are what I built on top after that line got crossed. That's where it got strange and good.
The real story is chaos (and a lot of neuroscience)
Not the tidy version a stats line implies. The honest version is that a good chunk of this wasn't writing code at all — it was me down a rabbit hole reading about how human memory actually works. Working memory versus long-term. Why episodic and semantic are genuinely different systems in your head and not one blurry bucket. How the brain decays, consolidates and promotes — what it keeps, what it bins overnight, what it drags up years later. I'd surface from a paper on memory consolidation, go re-architect a layer, break three others, and dive back in. The five-layer model didn't come off a whiteboard. It came out of a fortnight of reading neuroscience and repeatedly going wait — I'm modelling this completely wrong.
The commit graph just wears that honestly. 413 commits and 91 migrations at the time of writing, one author, October to now — shaped exactly like obsession looks: furious multi-week bursts where I couldn't leave the thing alone, bracketing dead stretches where a hard problem, or doubt, or just life parked it for weeks. Nobody builds something like this in a straight line. You build it in fits, in the gaps, past midnight, chasing an idea that won't let go — until one day it stops being an experiment and it's just real.
October — 103 commits. I turned the engine into a product. Preferences with temporal decay, tagging, cross-workspace search, a browser extension, and one productionisation day where I landed a 272× performance win on local ONNX embeddings (210ms → under 1ms) — later superseded by the measured cloud-embedding win you'll meet in Bet One, but it made the thing usable at the time. Same day I shipped multi-tenant workspaces with RBAC and API keys and stood up the first Agent SDK. It went from "a memory engine" to "a thing teams can log into and pay for."
November–December — four commits, total. The pause. An RFC and a refactor checkpoint. Not shipping — thinking. This is the beat I'd keep front and centre, because it's where I made the call that changed everything: the pivot from "memory dashboard for humans" to "decision substrate for agents." Deliberate, in a quiet winter, not stumbled into. The whole centre of gravity of the project moves in those four commits.
January through March. I built the machinery that pivot needed. I graduated the semantic layer from Postgres relationships to a real bi-temporal knowledge graph — first on Apache AGE as a stepping stone, then onto Neo4j/Graphiti (I've since deleted the AGE and Postgres graph tables outright; Neo4j is the sole semantic store now, cutover done). An execution-memory layer so agents persist their working state. SDKs in Go and Python. Editor plugins for Claude Code, Cursor and OpenCode. Then the developer-platform layer — OAuth, webhooks, scopes, a security audit.
There's a not-every-call-sticks beat in here too, and I'll tell it because build-in-public means telling it: I pivoted auth from Clerk to self-hosted Logto. The U-turn landed cleanly — Logto is live in production today at auth.dangerouslyskip.com, and the only Clerk residue left is a fossilised field name (req.clerkUserId) I haven't bothered to rename. Not every pivot resolves this well. This one did.
Then April and May go silent — one of those dead stretches. And June detonates: 78 commits, my biggest burst, where I stopped polishing "a better memory store" and started making actual bets. The migration numbers read like a plot: 001 (short-term table) → 026 (multi-tenant, the inflection) → 046–050 (decision graph + gate) → 067–069 (Project Hives), and they've kept climbing into the nineties since. You can watch the thesis form in the schema.
These were the four bets.
Bet one: measure it, or shut up about it
Before I trusted a single one of my own intuitions about retrieval quality, I built a LoCoMo eval harness — retrieval-recall plus QA-F1 with an LLM judge — and measured what actually moves the needle. The findings are my favourite kind, because they're humbling.
My clever ideas lost. Hybrid search, fact-extraction, HyDE — none of them helped. What helped was boring: a bigger embedding model (text-embedding-3-large, 3072-dimensional) plus a rerank stage. Recall went from roughly 70% to 80%. I shipped the winners and dropped the clever ideas.
The point isn't the ten points of recall. It's the discipline. After this I stopped asserting Rekall's memory quality and started measuring it (I added a second benchmark, LongMemEval, later). In a space drowning in "our memory is smarter" hand-waving, "I ran the eval and my clever idea lost" is the most trustworthy sentence I can write.
Bet two: the decision context graph (the flagship)
This is the one the whole thesis hangs on. Most memory tools remember what was said. I added a first-class layer for what was decided, why, and how it turned out.
I made decisions first-class nodes — a real choice, deliberately selected over alternatives — wired with a proper edge vocabulary: decided_by, chose_over, based_on_evidence, resulted_in, superseded_by, influenced. I pull them out of ordinary conversation with ambient extraction (a spike I ran showed this viable at near-100% precision even on a model 40× cheaper than frontier), keep them on a dedicated recall channel so they never pollute normal search, and — the part that turns it into a learning loop instead of an archive — let you record the outcome. How did that decision actually go? That writes a resulted_in edge, so the system can later tell you: you tried this; here's how it went; here's what superseded it.
That's continuity of judgment, concretely. An agent or a team walks the reasoning trail behind a past choice and sees whether it worked — instead of confidently re-making a decision someone already learned the hard way was wrong. A notepad cannot do this. It doesn't even know a decision happened.
Bet three: the validation gate — autonomy with brakes
A decision substrate agents can act on needs brakes, so I built a validation gate: a pre-action gate governing what an agent does (it never touches the read/store path — reading memory is always free). I gave it the rules a regulated environment actually cares about — respect_superseded, no_contradict_confirmed, require_confirmation_low_confidence, budget_limit, permission_required, cross_hive_share — promoted through a shadow → warn → enforce ladder so I can watch it be right before I let it block anything. Fail-open. Fully audited.
The detail I like best: a require_approval verdict used to be a dead end — a token that said "a human needs to look at this" and then lost the action. Now the held action is persisted, and approving it actually replays it. The brake doesn't drop the work on the floor. It parks it and resumes it on a nod.
That instinct should feel familiar — it's the one behind Echo's earned auto-approve and Governor's policy-as-code. Let the system act autonomously, inside inspectable guardrails, with a human routed in only where a human is genuinely required. Agents that act with brakes won't act against a confirmed decision, blow a budget, or leak across a tenant boundary without someone signing off.
Bet four: Project Hives — from remembering work to governing it
My newest June flagship reframes the scope again. If the decision graph is memory of judgment, Project Hives is the substrate that judgment runs on: a shared, governed place where AI-driven development actually happens. (And yes — confusingly — also "hives." The hive from up top is the shared-memory scope; a Project Hive is the work layer. Same word, two jobs. I never claimed my naming was disciplined.)
The framing rule I set myself is the tell: the coding agent already decomposes a goal into a plan. My job isn't to re-decompose it — it's to capture that decomposition and de-silo it. Explicitly: don't copy Jira. This isn't a ticket tracker. It's an attribution-and-provenance layer for the AI software-development lifecycle — who (which human, which agent) did what, when, for how long, ideally with token cost — so the work becomes shared, queryable memory.
Underneath, the parts I'm proudest of:
- The spine (migrations
067–069): projects inside a tenant; tasks with a real lifecycle (backlog → ready → in_progress → in_review → blocked → done → cancelled); a task-dependency DAG with a recursive-CTE cycle guard so the graph can't eat itself. - Race-free atomic claim. A swarm of agents pulls from one ready queue via
UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1)— zero double-assignment, and a task only becomes claimable when every dependency isdone. That one SQL clause is what lets many agents share a backlog without stepping on each other. - Leases, not locks. A claim carries a
lease_expires_at, and a sweep worker auto-requeues abandoned work — so a crashed agent doesn't wedge a task forever. Assume the worker dies; design so the work survives it. - The work-event log — the actual point. An append-only
work_eventstable records every claim, status change and completion with duration, principal and source. That's the raw material for an AI-SDLC report: who did what, how long it took, the human-versus-agent split. Provenance, not project management. - Ambient capture — the zero-effort path. My favourite is an
ExitPlanModehook: when an engineer in Claude Code exits plan mode, aPostToolUsehook ships the emitted plan straight into the bound project as backlog tasks. The engineer does nothing. The decomposition de-silos itself.
That last one is the whole philosophy in a single feature. The plan a developer made was always going to evaporate into one person's session. My hook catches it on the way out and turns it into shared, attributed, resumable memory — automatically. Which is the exact argument I make in Human in the Loop Is a Trap: work shouldn't live in a single-player silo waiting to be copy-pasted; it should land in shared memory the moment it exists. This is that essay's argument shipped — memory as the runtime, the turn-based silo replaced by a substrate everyone reads and writes.
July: it's alive — and there's a fourth door
I ended the first version of this post on "June detonates" and "pre-users." Then July shipped the most demo-able chapter of the whole thing, so I owe the snapshot an update.
Agents that coordinate over an open protocol. I shipped an A2A adapter — agent-to-agent over the open A2A spec: JSON-RPC, SSE streaming, push notifications, wake-on-message. This is the strongest proof of my substrate thesis I've got. It's not "my agents share my memory store"; it's agents coordinating over shared memory across the open standard, waking each other on message. My multiplayer argument stopped being a slide and became a wire protocol.
Husk agents — the fourth door. Remember that "what's left" line about retiring the legacy agent model onto the unified an agent is its memories constellation? I finished it — and it turned into something much bigger than a cleanup. Husk agents: immortal, resurrectable agents I can rebuild entirely from their stored memories; a specialist bench with an expertise graph; agents that outlive the person who created them and survive an offboarding. An agent whose whole working self is memory can be paused, moved, killed, and brought back — or inherited. That's not a memory feature. It's a fourth door I didn't design so much as discover.
And it's actually running. Plus the connective tissue that drags a research build into product territory: an agent observability UI; skills, rules and goals dashboards with webhooks; quotas and token accounting; and hive-org security hardening where I closed real cross-tenant leaks. The headline I care about most: it's deployed and live, on self-hosted CI, inside an org repo. My honesty section below used to say "research-grade and not yet deployed." Half of that is now false — it's deployed and running. The other half stands, and I'll keep it honest.
The maturity turn, which is also the whole metaphor
One movement in this stretch wasn't a feature. I'd been letting the CI type-check and lint steps run informational — failures swallowed. When I finally turned them into hard gates, I had to pay down the debt behind them first — and that debt turned out to be hiding real, shipping bugs the type checker had been flagging at me all along: a graph client calling an embedding method that didn't exist; three MCP tools wired against an imagined service API that threw on every single call; a status check compared against a value that never existed; an agent-memory route silently dropping the filter that was supposed to scope results to one agent. I drove tsc from 110 errors to 0, and it surfaced and fixed each one. (ESLint went 135 → 0 too, but that was mostly dead code — low-stakes, about making the gate trustworthy, not runtime risk.)
The meta-point is too on-the-nose not to say out loud: the type checker became a collaborator that finds bugs the moment I trusted it instead of silencing it — which is exactly what I'm selling with Rekall. Continuity of judgment beats a louder notepad, even when the judgment in question is my own compiler telling me the truth for months while I ignore it.
The honest reality check
Straight, because the whole pitch is honesty: my differentiating bets are still pre-users. The decision graph, the gate, Project Hives, husk agents — deployed and running now, live in an org repo, but not yet validated by real people relying on them. They're a placed bet to differentiate, not a proven, adopted product. The competitive frame I'm up against is mem0 / Zep / Letta / Cognee / hivemind, and the two flagship gaps I'm betting on are skill codification and the decision graph. Whether those are the right bets is exactly what real users will tell me and my intuition won't.
What I've still got left, in priority order: prove the decision graph and Project Hives with real people relying on them (this matters more than anything — an unvalidated bet is just a nicely-typed opinion); publish the framework adapters to npm and PyPI (until I publish them, "the memory backend for every agent framework" isn't actually installable); and finish the Project Hives external sync (two-way Jira/GitHub, plus a live TodoWrite → task status feed straight from the agent). The husk-agent / constellation migration that used to sit on this list is done.
Why it matters
As agents take on more autonomous work, "did it remember the fact?" stops being the interesting question. The interesting one becomes "did it remember the judgment — what we decided, why, how it turned out — and will it act in line with it?" A notepad can't answer that, no matter how big you make it. A governed decision substrate can. And when those agents can wake each other over an open protocol and outlive the people who built them, the substrate stops being a nice-to-have and becomes the thing the whole system stands on.
That's the bet. Not a better memory for AI. A memory with judgment attached — and the brakes to honour it.
You can build a bigger notepad. It still won't learn. This is the other thing — and I'm building it.
Companion reading: the thinking here runs through the essay series The Harness-Harness — most directly in Your Harness Has Amnesia, where Rekall is the argument shipped.