Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Abstract: AI agents increasingly operate over extended time horizons, yet their ability to retain, organize, and recall multimodal experiences remains a critical bottleneck. Building effective lifelong memory requires navigating a vast design space spanning architecture, retrieval strategies, prompt engineering, and data pipelines; this space is too large and interconnected for manual exploration or traditional AutoML to explore effectively. We deploy an autonomous research pipeline to discover Omni-SimpleMem, a unified multimodal memory framework for lifelong AI agents. Starting from a naïve baseline (F1=0.117 on LoCoMo), the pipeline autonomously executes ${\sim}50$ experiments across two benchmarks, diagnosing failure modes, proposing architectural modifications, and repairing data pipeline bugs, all without human intervention in the inner loop. The resulting system achieves state-of-the-art on both benchmarks, improving F1 by +411% on LoCoMo (0.117$\to$0.598) and +214% on Mem-Gallery (0.254$\to$0.797) relative to the initial configurations. Critically, the most impactful discoveries are not hyperparameter adjustments: bug fixes (+175%), architectural changes (+44%), and prompt engineering (+188% on specific categories) each individually exceed the cumulative contribution of all hyperparameter tuning, demonstrating capabilities fundamentally beyond the reach of traditional AutoML. We provide a taxonomy of six discovery types and identify four properties that make multimodal memory particularly suited for autoresearch, offering guidance for applying autonomous research pipelines to other AI system domains. Code is available at this https://github.com/aiming-lab/SimpleMem.

• The paper introduces the Omni-SimpleMem framework, which uses an autonomous research pipeline to discover and optimize agent memory architectures for multimodal tasks.
• The paper demonstrates state-of-the-art improvements with a +411% F1 gain on LoCoMo and a +214% F1 increase on Mem-Gallery benchmarks via iterative experiments.
• The paper highlights that code-level bug fixes, architectural reworking, and prompt engineering can yield disproportionate performance gains over traditional hyperparameter tuning.

Omni-SimpleMem: Autonomous Discovery of Lifelong Multimodal Agent Memory

Introduction

"Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory" (2604.01007) presents an autonomous research methodology for systematically discovering, optimizing, and validating memory architectures for AI agents operating across diverse multimodal long-horizon environments. Leveraging the AutoResearchClaw pipeline, the study rigorously explores the architectural, algorithmic, and operational design space of agent memory—encompassing storage substrates, retrieval mechanisms, prompt strategies, and data handling—beyond the limited reach of traditional AutoML. The resultant framework, Omni-SimpleMem, achieves state-of-the-art results on prominent memory-centric benchmarks, LoCoMo and Mem-Gallery, surpassing both naive and prior SOTA baselines by substantial margins.

The core contribution lies not only in the demonstrated empirical gains but in the meta-algorithmic process—a ~50-experiment inner research loop run autonomously, diagnosing and repairing bugs, proposing and testing architectural changes, and modulating system behavior across data curation, storage, and retrieval. This work substantiates the value of autoresearch in high-dimensional, tightly coupled AI system domains.

Figure 1: The discovery process of Omni-SimpleMem: (a) Architectural overview—multimodal inputs undergo novelty filtering, MAU compression, and retrieval via hybrid dense-sparse-graph search with pyramid expansion; (b) Autonomous optimization trajectory on Mem-Gallery, showing a +214% F1 improvement over 39 experiments.

Autonomous Research Pipeline

AutoResearchClaw, a 23-stage research pipeline, orchestrates the end-to-end cycle from experimental hypothesis generation through code modification, evaluation, error diagnosis, and iteration. The pipeline initiates from SimpleMem—a unimodal, text-centric framework—and coordinates LLM-enabled agents to diagnose system behavior, propose architectural or prompt-directed modifications, implement targeted code changes, and decide on continuation or reversion based on precise metric improvements ($\Delta$F1 $\geq 0.5\%$).

This agent-driven process incorporates granular error classification (including API, dependency, runtime, and format errors) and autonomous repair mechanisms, e.g., switching to alternative embedding models upon upstream API failure. Notably, the highest-impact improvements manifested not in parameter tuning but in code-level bug fixes, architectural reworking, and context/prompt engineering, each with independently quantifiable effects exceeding cumulative hyperparameter optimization.

Figure 2: Optimization trajectories for LoCoMo (top: 9 iterations) and Mem-Gallery (bottom: 39 experiments, 7 phases); key discoveries and performance jumps are annotated, with accepted, reverted, and failed experiments clearly delineated.

Omni-SimpleMem Architecture

The final discovered architecture exhibits three central principles:

1. Selective Ingestion:

Modality-specific novelty detectors (CLIP for vision, VAD for audio, Jaccard filtering for text) detect and discard redundant or low-information content. Retained signals are encapsulated as Multimodal Atomic Units (MAUs), which house summaries, embeddings, and metadata in hot storage and raw assets in cold storage.

2. Progressive Hybrid Retrieval:

Retrieval is realized via hybrid dense (FAISS-based ANN over MiniLM or text-emb-3-large vectors), sparse (BM25 keyword match), and graph-based (knowledge graph expansion) mechanisms. The system employs a set-union merging rule, favoring dense semantic ordering and appending BM25-only hits, as discovered by the pipeline. Information is loaded in a pyramidal sequence—compact summaries are provided initially, with expansion to detailed text or raw assets contingent on query-context token budgets.

3. Knowledge Graph Augmentation:

MAUs are linked into a dynamically constructed knowledge graph, enabling relational and multi-hop reasoning. Entity resolution employs combined embedding and string-based similarity heuristics, and query-time hops over the entity-relation topology augment retrieval sets with relationally connected facts, each scored with a decay function proportional to graph distance.

Figure 3: Omni-SimpleMem system: (Left) selective multimodal ingestion, (Center) MAU hot/cold storage and knowledge graph construction, (Right) retrieval via hybrid search and hierarchical pyramid expansion, all gated by token budget $B$.

Empirical Results

Benchmarks and Baselines

Omni-SimpleMem is evaluated on LoCoMo and Mem-Gallery, which require reasoning over long-term, multimodal conversational histories with associated QA tasks. Six baselines including MemVerse, Mem0, Claude-Mem, A-MEM, MemGPT, and SimpleMem are used for comparison, spanning architectures leveraging text-only storage, explicit memory management, and knowledge-augmented retrieval.

Trajectory and Discovery

On LoCoMo, nine optimization iterations (plus reversions) yielded a +411% F1 jump (0.117→0.598), the leading improvement arising from correcting an output verbosity bug (missing response format), detailed MAU timestamp repairs, and the adoption of set-union dense/sparse hybrid search. Mem-Gallery optimization involved 39 experiments in seven discrete phases, ultimately improving F1 by +214% (0.254→0.797); the largest single gain followed the shift from summary-based to full-text retrieval for evaluation compliance.

Main Numerical Results

Omni-SimpleMem consistently outperformed all baselines across LLM backbones (GPT-4o to GPT-5.1), e.g., F1 up to 0.810, with prior SOTA trailing by >25 percentage points. Ablation studies confirm that the most significant components (pyramid expansion, BM25 hybrid search) directly accounted for the largest performance penalties when removed.

In addition to accuracy, Omni-SimpleMem demonstrates high throughput, with 3.5$\times$ speedup over the fastest baseline (up to 5.81 queries/sec with 8x parallel retrieval workers—parallelizable due to the decoupled nature of FAISS/BM25 search versus LLM generation).

Architectural Analysis and Implications

Autonomous optimization revealed that in agent memory systems:

• Bug fixes and data pipeline repairs can yield disproportionate performance improvements (up to +175%), highlighting the limitations of approaches restricted to hyperparameter tuning.
• Prompt positioning and constraint injection produce significant metric swings (notably +188% on select Mem-Gallery categories), underscoring the necessity for joint architectural-prompt codebase search.
• Hybrid retrieval (dense/sparse/graph) and progressive pyramid expansion ensure high-precision retrieval while managing context length limitations and budget-constrained inference.

These findings illustrate that auto-research is best suited for system domains with: (1) immediate, scalar reward signals; (2) modular, revertible architectures; (3) short enough iteration cycles for inner-loop optimization; and (4) full provenance and control over system state/outputs.

Future Directions

This study demonstrates the feasibility of fully-autonomous, iterative agent-driven research in complex, multimodal AI system domains, with implications for rapid, unbounded exploration of the AI systems design space. Potential directions include deployment for other tightly coupled or multi-component domains (e.g., tool use, planning, multi-agent collaboration), as well as meta-optimization of the research strategy itself (e.g., adaptive allocation of search effort, cross-pipeline coordination).

Conclusion

Omni-SimpleMem, autonomously discovered through the AutoResearchClaw pipeline, sets a new state-of-the-art for lifelong multimodal agent memory. The high-impact discoveries—architectural, prompt, and code-level—reveal that system-level autoresearch offers optimization powers fundamentally exceeding traditional AutoML, especially for domains where system bugs, data semantics, and component interactions are tightly entangled. The combination of reproducibility, rapid iteration, and modular evaluation suggests a promising paradigm for future AI system development and evaluation.