Building personal memory store

When I played with Grok Bot, I was hooked by how proactive and useful it could be. It's an impressive product, both in its capabilities and its UI. But it didn't know me, my life context, or what I care about right now. It was cool, but I'd never call it a personal butler that can help me with all my daily chores and projects.

Intelligence is only one part of what makes AI agents useful. The other part is context. Usefulness = intelligence + context.

We are fortunate to have many intelligence providers, with models getting more powerful and more affordable every few weeks. And this is precisely why it makes little sense to keep all my personal context with any one model provider. I want to be able to switch between models without being locked into any of them, and own my personal context and have full control over it.

When I started designing my little village of AI agents, one of the first decisions I made was to build a decentralised memory store that I could manage myself and connect to any third-party model or agent. It's been a fun journey, and in this post I'll describe how I built it.

This is Part 1 of my Personal Little AI World setup series.

Architecture

I use open-source Graphiti framework and FalkorDB to orchestrate my memory store. Graphiti parses information and stores context as relationships between entities: "Entity → Relationship → Entity".

With more queries the graph grows and develops more complex relationships like "Oleh friends with James who works at XYZ competitor of Acme".

The good news is that Graphiti abstracts away most of the complexity for you - entity deduplication, query parsing, conflict management, etc. The framework has been great at hiding the complexity while still being developer-friendly enough for me to experiment with custom entity types and search strategies.

Graphiti supports several graph databases out of the box. I went with FalkorDB because it's the simplest option and works well at my small, personal scale.

Alongside context, I want my memory store to store my todos and reminders. Vector DBs are a bad choice for storing tabular data, as similarity search can't precisely answer questions like "what's due before Friday?". To store tabular data, I added a SQLite db and a thin piece of logic that parses free-text todo queries into SQL rows.

I expose memory store data via an MCP server for agents and a REST API for analytics for humans.

When an agent decides to add a new memory, it calls add_memory with the new context as a function parameter. I always store an input query (Graphiti uses the "Episode" term for raw text) in the Episode store. This means I always have a source of truth for everything further down the pipeline. Then I run an LLM classifier call to decide whether the Episode should be added to the Vector DB, stored inside SQLite, or both.

For search, I expose a search_memories function that performs similarity search across my context and returns a list of relevant facts, entities, and source-of-truth Episodes. For Todos, I decided to simply attach a list of active TODOs to the response. It's not the most elegant solution, but it works really well at my scale and keeps things simple.

<FACTS>
{to_prompt_json(fact_json)}
</FACTS>
<ENTITIES>
{to_prompt_json(entity_json)}
</ENTITIES>
<EPISODES>
{to_prompt_json(episode_json)}
</EPISODES>
<Todos>
{to_prompt_json(active_todos)}
</Todos>

Markdown first or graph first?

There are two schools of thought when it comes to storing personal context.

Markdown first (e.g. gbrain): keep the source of truth as Markdown files and give agents access to a filesystem, with (optional) graphs built on top.

The biggest benefit is that you can easily edit files, read them on demand, and store text artefacts (e.g. emails) in their original form. You can always go back to the source of truth.

The downside is that you don't get the transactional guarantees of a database, you or your agent need to manage a library of files, and as it grows, you also need an indexing/search layer rather than scanning Markdown files directly.

Graph first: store everything as a graph, linked back to the parent Episode, and rely on a combination of vector, keyword, and graph search to retrieve relevant context. You can also generate Markdown files from the graph on demand.

The downside is that a graph is harder for a human to read and edit directly. Once you turn an email or conversation into entities and relationships, you lose some of the original context. You might know that "Oleh works at Acme", but the context behind that relationship is stored in the parent Episode. You have to explicitly trace the relationship back to the Episode to understand where that fact came from.

I don't think one approach is obviously better for a personal context store. My decision to build graph first system came down to two things: 1) Graphiti works really well with this model and 2) I'm very unlikely to successfully maintain a Markdown-based memory system, because... I'm lazy.

Conflict management

What happens if two contradictory facts come into your memory store? Graphiti uses timestamps to figure out which fact is newer and treats the newer fact as replacing the old one. So if last season you supported Arsenal and now you're a big City fan, the newer fact will supersede the old one.

In my system, it's the user's (aka my) responsibility to resolve contradictions. The system will highlight when two facts can't coexist, and it's up to the user to decide which colours they want to support this season.

Deployments

I deploy the memory store and my fleet of agents to exe.dev. The exe model is simple, yet unusual: you pay a monthly fee for a pool of CPU and memory, then split it across as many VMs as you like at no extra cost per VM.

Almost unlimited VMs, plus the extremely convenient abstractions around authentication and integrations make the platform perfect for my little AI world. Exe is just such a great example of elegant ergonomics and dev-friendly design, so this post will include several love notes to the platform. (unfortunately, this post is not sponsored by exe)

Backups

Don't forget to back up your memory store! In my case, it's a simple cron job that pushes a db dump to a 3rd party storage provider. Do whatever makes most sense for you.

Connect Claude to my personal memory store

First ode of love to exe.dev! My memory store is a VM running on the exe platform. Each VM is private by default, but conveniently exposes a URL. For my memory store, it's memory-store.exe.xyz (unless you're logged into my account, you'll see a 307 redirect).

Exe also allows me to generate a bearer token and securely send data to the VM. It also manages port forwarding between the public URL and the FastAPI MCP server running on the VM.

Add a new Connector to Claude and add an Authorization header with your token.

Claude can now add and search my memories!

Pro tip: from my experience, Claude sometimes needs an extra nudge to actually use these tools. Anthropic allows you to add instructions for every conversation. I'm still figuring out the best wording.

Connecting memory store to my fleet of Agents

Second ode of love to exe.dev! I'll talk about my agent fleet in more detail in Part 2 of this series, but as TLDR, each Agent runs in a separate VM on the exe platform. I can tag each VM and then create rules like "allow each VM tagged agent to access the memory store".

This is incredibly sleek! I don't have to worry about auth, don't store any keys inside the VM (with a running Agent inside it!) and each new VM with tag agent talks to the memory store out of the box! How cool is that!

Is it useful?

Million dollar question: does having a personal (sovereign!) memory layer and connecting it to Claude, Pebble smart ring and a fleet of agents make my life easier, more productive and/or more enjoyable?

Yes, sort of, maybe, it depends! Voice notes over pebble smart ring (more in part 2) to save todos works really well, especially if I add a deadline for it "Remind me to do x on Sunday".

Some of the facts derived from my interactions with agents and stored in my memory store are genuinely useful and correct. But I wonder: will they ever actually be useful in the future?

And some of the facts, especially those derived from my voice, are completely nuts.

I don't know who Jeff is, and I've never had coffee with Sheetal :(

My hope and the promise of personal memory overall is that, over time, it builds up a critical mass of knowledge that becomes invaluable to an agent trying to resolve any personal matter.

I can already see glimpses of it, but I currently still need to babysit extraction and remove "Jeff had coffee with Sheetal" nonsense facts. I reckon in a few months it will be clear if this whole setup actually worth it. And whether it works or doesn’t, that could make for a pretty good story to tell in Part 3 of this series.

Teaser for Part 2

• Deploying Hermes agents with custom plugins
• How I use my smart ring repebble.com
• Building my own Grok Bot alternative with Rust, TypeScript and Tauri
• More love to exe.dev