Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Key points

• Problem: AI agents that work over long horizons are bottlenecked by memory for storing and recalling text, image, audio, and video experience, and building good memory means searching a large design space where architecture, retrieval strategy, prompts, and data pipelines interact. Existing memory systems come from manual research cycles in which a single researcher may explore only a handful of configurations per day. Traditional automated machine learning (AutoML) searches predefined numerical hyperparameter spaces, leaving code comprehension, bug diagnosis, and architectural redesign outside its reach.
• Input and output: This is a system paper about a memory system whose design was found by an autoresearch pipeline. The 23-stage pipeline AutoResearchClaw took the code of the text-only memory SimpleMem, two benchmark evaluation harnesses that report F1, and LLM API access, and produced Omni-SimpleMem. Omni-SimpleMem stores text, image, audio, and video inputs as memory and answers a query by putting the retrieved memories into the context.
• Method: The pipeline analyzes prior results, forms a hypothesis, changes the code, and evaluates it on a benchmark; it proceeds when F1 improves by 0.5% or more, refines the hypothesis when the result is ambiguous, and reverts and pivots after two consecutive drops. The discovered architecture filters redundant inputs by comparing CLIP image embeddings across consecutive video frames, voice activity detection (VAD) probability for audio, and Jaccard overlap with recent summaries for text, and stores the rest as Multimodal Atomic Units (MAUs), keeping summaries and embeddings in hot storage and raw content in cold storage. Retrieval combines set-union merging that keeps the FAISS dense vector ranking and appends BM25 keyword results, pyramid retrieval that starts from summaries of top candidates, opens full text when similarity exceeds a threshold, and loads raw content within a token budget, and knowledge graph expansion over 7 entity types.
• Implementation: Omni-SimpleMem is implemented in 13,300 lines of Python across 11 subpackages, and its default build uses dense retrieval with FAISS and all-MiniLM-L6-v2 embeddings (384 dimensions), visual novelty checks with frozen CLIP ViT-B/32, and knowledge graph extraction with GPT-4o in JSON mode. Defaults are top-k 20, a Level 2 expansion threshold of 0.4, and a token budget of 6,000, while for LoCoMo the pipeline chose text-emb-3-large embeddings and top-k 20 to 30. The README of the official GitHub repository bundles SimpleMem, Omni-SimpleMem, and EvolveMem into one simplemem package that picks the Omni backend when a multimodal input function is called first, and points to LoCoMo and Mem-Gallery reproduction scripts in the OmniSimpleMem folder and a Model Context Protocol (MCP) server that supports text memory only.
• Results: In about 72 hours the pipeline ran about 50 experiments, raising LoCoMo F1 from the initial 0.117 to 0.598 (+411%) and Mem-Gallery F1 from 0.254 to 0.797 (+214%). The biggest LoCoMo gain (+175%) came from finding a one-line bug where the API call lacked response_format, and on Mem-Gallery returning the full original dialogue text instead of summaries gave the largest gain of +53%. With all 5 LLM backbones, Omni-SimpleMem's overall F1 was higher than all 6 baselines, from 0.492 (GPT-4.1-nano) to 0.613 (GPT-5.1) on LoCoMo against 0.342 to 0.432 for the previous best SimpleMem, and from 0.749 to 0.810 on Mem-Gallery.
• Limits: The +411% and +214% gains are relative to the initial configurations; the first LoCoMo improvement fixed a missing response_format bug in the starting configuration, and the first Mem-Gallery phase upgraded the LLM from gpt-4.1-nano to gpt-4o and moved to local embeddings (+39%). Humans supplied the starting code, the two evaluation harnesses, API access, and the instruction to extend to multimodal memory, and evaluation used the LoCoMo and Mem-Gallery benchmarks. The authors name the conditions under which this approach fits: immediately computed scalar metrics, a modular architecture, short cycles of 1 to 2 hours per experiment, and version-controlled code changes that can be reverted.

Paper links

External research summaries. These are not HDATF publications or measured product results.