Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

Key points

Standard retrieval-augmented generation does not fit agent memory well.
Unlike large heterogeneous corpora, agent memory forms a bounded and coherent interaction stream in which many spans are strongly correlated or nearly redundant.
As a result, flat top-k similarity search often returns redundant context, while summary-centric hierarchies blur the subtle details that distinguish one candidate from another.
The paper argues that agent memory should follow the principle of separation before aggregation, meaning the system should first separate reusable facts, updates, and distinguishing details from similar histories before organizing them for efficient retrieval.
Based on this principle, the authors propose xMemory, which builds a mutable hierarchical memory structure from original messages to segments, memory components, and groups.
xMemory splits interaction history into local events, separates each segment into memory components, aggregates relevant components into higher-level groups using sparsity and semantic fidelity objectives, and preserves this structure incrementally as memory evolves.

External research summaries. These are not HDATF publications or measured product results.