Memory Systems — Short, Long, Episodic, Semantic

Agent memory is just context that survives across turns. Four useful categories:

1. Short-term (working) memory

• The current conversation context.
• Bounded by the model's context window.
• Mechanically: the message list passed in each LLM call.
• Failure mode: drops once you exceed the window.

2. Long-term (persistent) memory

• Facts you want to survive sessions.
• Stored in a DB (often vector + metadata) and retrieved at conversation start.
• Examples: user profile, preferences, past decisions.
• LangMem / Mem0 / Letta abstract this; rolling your own is fine.

3. Episodic memory

• Past conversations themselves, retrievable later.
• Indexed by content (vector) + metadata (date, topic).
• Lets the agent recall "we discussed X last Tuesday".

4. Semantic memory

• Domain knowledge, not user-specific.
• This is your RAG corpus.
• Same retrieval mechanics; conceptually distinct purpose.

Layered architecture

┌────────────────────────────────────────┐
│ Working memory (current conversation)   │  ← message list
├────────────────────────────────────────┤
│ Long-term: user profile / facts         │  ← key-value store + vector
├────────────────────────────────────────┤
│ Episodic: past conversations            │  ← vector store of summaries
├────────────────────────────────────────┤
│ Semantic: knowledge base                │  ← RAG over your docs
└────────────────────────────────────────┘

At each turn, the agent has hooks to query each layer. Good agents query selectively — pulling memory is a tool call, not always-on context.

Memory write patterns

• Auto-extract. A small LLM watches the conversation and writes "user prefers X" facts to long-term store.
• Explicit save. User says "remember that I'm allergic to nuts" → agent calls a remember tool.
• Summarize-and-store. At session end, summarize the conversation and store as episodic memory.

Memory pitfalls

• Stale memories. What was true last year is wrong now. Add timestamps and TTLs.
• Conflicting memories. "User loves coffee" and "user is caffeine-free" both stored. Resolve at retrieval, prefer most recent.
• Memory leak across users. Multi-tenant systems must scope memory by user/org.
• Over-retrieval. Pulling 20 memories per turn dilutes the prompt. Top-3 with scoring usually wins.

Practical stack

• Working: in-process message list.
• Long-term: Postgres (key-value or JSONB) + vector index (pgvector).
• Episodic: Vector DB (Qdrant, Weaviate) of conversation summaries.
• Semantic: your RAG stack.
• Layer abstraction: Mem0, LangMem, or roll your own.