A stateless agent treats every interaction as if it has never met the user before. It cannot remember previous conversations, learn from past mistakes, or personalise its behaviour. Memory is what transforms a tool into an assistant. But memory in agents is architecturally complex: it requires storage, retrieval, relevance scoring, and privacy controls. Get it wrong, and your agent either remembers nothing (useless) or remembers everything (expensive and slow).
The Three Memory Types
(1) Short-term memory (conversation context): the current conversation's history. Stored in the LLM's context window. Limited by token count. Automatically available during a single session. (2) Long-term memory (persistent facts): information the agent learns about the user or domain across sessions. Stored in a vector database or key-value store. Retrieved via semantic search when relevant. (3) Episodic memory (past experiences): specific past interactions that are relevant to the current request. Stored as indexed conversation segments. Retrieved when the agent encounters a similar situation. Each type serves a different purpose and has different storage and retrieval requirements.
Long-Term Memory Architecture
Long-term memory requires: extraction (identifying which facts from a conversation are worth remembering), storage (writing facts to a persistent store with metadata — user ID, timestamp, source conversation), retrieval (finding relevant memories given a new request, using semantic search), relevance scoring (ranking retrieved memories by relevance and recency), and forgetting (implementing memory decay — older, less-referenced memories lose relevance over time). The retrieval step is critical: returning too many memories overwhelms the LLM's context; returning too few misses relevant context.
Note
The retrieval quality determines memory effectiveness. A vector database with good embeddings and relevance scoring is the foundation. Bad retrieval makes memory worse than no memory.
Privacy and Consent
Agent memory raises privacy concerns: what does the agent remember, who can access it, and how long is it retained? Implement: user consent (explicit opt-in for long-term memory), memory transparency (users can see what the agent remembers), memory deletion (users can delete specific memories or all memories), data isolation (memories from different users are never mixed), and retention policies (automated deletion after a configurable period). These are not nice-to-haves — they are compliance requirements for GDPR and CCPA.
Conclusion
Agent memory transforms a stateless tool into a personal assistant. The three memory types (short-term, long-term, episodic) serve different purposes and require different architectures. The key is retrieval quality and privacy controls — remember what matters, forget what does not, and give users control.
Key Takeaways
- Three memory types: short-term (context window), long-term (persistent facts), episodic (past experiences)
- Long-term memory: extract → store → retrieve → score relevance → forget
- Retrieval quality determines memory effectiveness — bad retrieval is worse than no memory
- Privacy is a compliance requirement: consent, transparency, deletion, isolation, retention policies
- Memory transforms a tool into a personal assistant — it is the key differentiator