Key Concepts for Implementing Memory in AI Agents

An agent without memory starts from scratch with each interaction and feels less intelligent. Memory helps an agent use context from earlier interactions. First, we need to define what type of memory we want to implement.

Types of Memory

Memory in agents borrows analogies from cognitive psychology. The Atkinson–Shiffrin memory model distinguishes memory stores; semantic, episodic and procedural memory are separate cognitive distinctions, not categories from that model.

• Short-term: Your brain’s notepad; in humans, it lasts very briefly between 0 and 20 seconds.
• Semantic: Essentially the meaning of the environment, facts, definitions.
• Episodic: Records of previous conversations: what was said, decisions made, important events.
• Procedural: Past instructions, behaviors learned through repetition.

State and Memory

“State” is one of the most relevant concepts in LangGraph: a LangGraph application is modeled as a State Machine. An agent without cross-session memory can still orchestrate tools and workflows.

Many LLM API calls are stateless: the application supplies the context needed for each invocation. Some services also maintain conversation state. The state layer is precisely how we introduce memory into the agent, whether temporary or persistent, short-term or long-term. One could argue that the agent’s state is in itself memory, but clearly it’s not sufficient.

Transient or Persistent?

• Transient: lives only during the current session; is deleted if the agent or process restarts. Useful for prototypes or agents that should not store data for privacy reasons.
  • Can be modeled as a notepad in application state; the context window is the bounded input supplied to the model, not a persistence mechanism.
• Persistent: survives restarts; it may be scoped to one thread or shared across conversations. Durability is distinct from scope and from the context supplied to the model. Systems and papers (e.g., Mem0) report improvements under their evaluation conditions, not guarantees for every application. (arXiv)

Three Key Questions

• What do we want to save: facts, conversation summaries, decisions, failures/successes.
• How we save it: relational database, vector DB, graph, or a combination. Do we use a tool or an asynchronous consolidation process.
• When we save it: immediately, at intervals, at the end of the session, or through asynchronous consolidation processes.

What We Want to Save

• User data (semantic): name, preferences, roles. These are “small” and stored as CRUD records—low cost and high precision.
• Long conversations (episodic): saving every token may not be useful. Instead:
  • Chunking + embeddings for semantic searches when you need to retrieve specific chunks.
  • Incremental summarization: at the end of N turns, generate a condensed summary that represents the essential part and store only the summary; keep expandable notes if reconstruction is required.
• Complex relationships: if you need to reason about relationships between entities (e.g., “customer X requested this in 2023 and repeated it in 2025”), consider a memory graph or relational representation that captures edges/properties. Some recent work proposes graph-based memory to capture temporal/causal relationships. (arXiv, ResearchGate)

How We Will Save It

There is no “correct database”; the decision depends on consistency, latency, cost, and audit requirements. A common pattern:

Working memory, scratchpad:

• Redis or an in-memory store for the last N turns; process-local storage is cleared on restart, while Redis durability depends on its persistence configuration.
• Or we can simply store this in the application state.

Semantic and fact memory:

• Document store / relational store for facts (e.g., Mongo, Postgres).
• Vector DB (Pinecone, Weaviate, Milvus, or Atlas Vector Search) for embeddings and similarity searches.

Consolidation, summarization, and reasoning:

• Background workers (cron, jobs) that: extract, condense, version, and write long-term persistent memory.
• Store in a vector store or knowledge graph.

Entities and relationships:

• In a graph with its own ontology, like neo4j

Memory layer as a service

• Projects like Mem0 deliver an “off-the-shelf” layer for dynamic extraction, consolidation, and retention policies; can reduce implementation work and operating costs. (mem0.ai, arXiv)

When to Save It

• Immediate: when a critical event occurs (e.g., preference change, purchase), the agent autonomously decides what and when to save.
• Turn counting: save only every N turns to avoid giving the agent the responsibility. We can keep a counter in the state that triggers the memory process.
• Idle-trigger: when you detect inactivity, consolidate the session.
• Batch/asynchronous: for long conversations, send background jobs that summarize and clean before persisting.

How We Will Consume It

• Load in system prompt: useful for small and critical data (user persona, constraints).
• Conditional retrieval: use signals (intent, slot-fill, topic change) to decide what memory to fetch.
• Tool-based access: the agent calls a “tool” (API) that returns relevant memories (instead of loading everything in the prompt). This separates responsibilities and enables caching, pagination, and authorization.
• Hybrid decision: load context summary + do additional retrieval when the response requires it.

MongoDB and other providers have published patterns and SDKs to enable cross-session memory and checkpointers in agent frameworks (e.g., LangGraph). If you work with LangGraph, check the available datastore/Store integrations to avoid reinventing the wheel. (MongoDB)

For a practical follow-up, see AI Memory Layer: Designing Persistent Memory for Agents.

Recommended Tools, Readings, and Resources

• Mem0 — memory as a layer (repository and docs). Worth evaluating if you’re looking for a memory solution with control and traceability; validate its behavior against your own requirements. (GitHub, docs.mem0.ai)
• Mem0 (paper/arXiv) — systematic evaluation and results on latency and token-cost. Useful evaluation context, not a substitute for testing your own workload. (arXiv)
• Hugging Face — posts on memory and how to reduce memory usage in models — good conceptual context and optimization patterns. (Hugging Face)
• Memory Architectures in Long-Term AI Agents (research) — paper on integrating episodic/semantic/procedural memory inspired by human cognition. Good theoretical support. (ResearchGate)
• MongoDB + LangGraph — examples and integrations (MongoDB Store for LangGraph) if you work with LangGraph and are looking for a scalable persistent memory solution. (MongoDB)