Mi-Memory: A Lifecycle Memory Framework for Personal AI

Abstract: Personal AI is moving beyond chat-only interaction toward continuous services that span phones, cars, homes, wearables, cameras, and tools. In this setting, memory cannot remain a cache of prior conversations. It should serve as a continuity and governance substrate: preserving durable user state, grounding answers in multimodal and device evidence, supporting correction and forgetting, bounding policy evolution, and remaining deployable under latency, cost, privacy, and edge-cloud constraints. This technical report presents Mi-Memory, a lifecycle memory framework for Personal AI organized around four roles: Structure, Expansion, Evolution, and Deployment. A shared audit contract links these roles through four recurring artifact families: typed evidence payloads preserve source identity and provenance, diagnostic traces localize evidence loss across the serving pipeline, strategy artifacts make memory-policy changes explicit, and gate/rollback records bound accepted evolution. MiMemory instantiates the roles through MemStack, MemSense/MemFuse, D$<sup>{2}$ACCI/E$<sup>{2}$MEND, and LiteMem. In controlled-reference Structure evaluations, MemStack reaches 93.59%, 57.24%, and 87.47% on LoCoMo, PersonaMem-V2, and LongMemEval, respectively; other tracks report module-level, preliminary/internal, transfer-feasibility, or design-only evidence with explicit boundaries. MiMemory is a step toward auditable, evidence-gated, and deployment-aware memory systems for Personal AI. Project homepage: https://darwin-agent.github.io/Mi-Memory/ .

• The paper introduces a lifecycle audit contract that links typed evidence, diagnostic traces, versioned strategies, and rollback records across memory structure, expansion, evolution, and deployment.
• The framework reports 93.59% on LoCoMo, 57.24% on PersonaMem-V2, 87.47% on LongMemEval, and a governed E2MEND improvement from 75.58% to 94.74% on LoCoMo.
• The results show practical transfer to lightweight deployment, with LiteMem retaining 90.0% of service-side improvement, while privacy enforcement, deletion propagation, scaling, and joint end-to-end ablations remain open challenges.

Overview and Motivation

This technical report presents Mi-Memory, a lifecycle memory framework for personal AI that treats assistant memory as auditable infrastructure rather than a static store or retrieval layer. The authors—Darwin Agent Team—argue that memory in multi-device personal AI (spanning phones, cars, homes, wearables, cameras, and tools) must satisfy obligations beyond long-term recall: preserving durable user state, grounding answers in multimodal and device evidence, supporting correction and forgetting, bounding policy evolution, and remaining deployable under latency, cost, privacy, and edge–cloud constraints.

The framework is organized around four lifecycle roles—Structure, Expansion, Evolution, and Deployment—linked by a shared audit contract carried through four artifact families: typed evidence payloads (source identity and provenance), diagnostic traces (stage-local localization of evidence loss), strategy artifacts (explicit, versioned policy changes), and gate/rollback records (bounded acceptance of evolution). Concrete modules instantiate each role: MemStack for structure, MemSense/MemFuse for expansion, D2ACCI/E2MEND for evolution, and LiteMem for deployment.

Problem Formulation

The report identifies four gaps that share a common cause—memory treated as a static store rather than a lifecycle. The state gap concerns incompatible granularities of durable facts, summaries, profiles, and sessions; the source gap concerns heterogeneous indexing across dialogue, images, tool traces, and device events; the governance gap concerns the inability to localize failures to ingestion, fusion, retrieval, filtering, assembly, or generation; and the deployment gap concerns cost, latency, privacy, and substrate assumptions.

Formally, at turn $t$ the system maintains a typed observation pool $O_t$, memory state $M_t$, selected evidence $E_{q_t}$, bounded context $C_{q_t}$, and answer $a_t$. The audit contract imposes three operational requirements: evidence preservation (traceable links from context back to sources), stage-local diagnosis (identifying where evidence was lost), and auditable evolution (hypothesis-tied, harness-evaluated, reversible policy changes). A key conceptual separation distinguishes serving correctness from update correctness—a correct answer can still hide an unsupported memory write or policy change.

Structure: MemStack

MemStack provides a typed, multi-granularity runtime with four layers: L0 atomic facts, L1 session/topic rollups, L2 stable profile entries, and SM session memory. Retrieval combines semantic vector search, lexical BM25, and LLM subquery expansion via weighted reciprocal rank fusion ($\kappa = 60$ default). Context assembly reserves high-priority slices for profile corrections and forget/update constraints, records overflow traces rather than silently dropping constraints, and keeps benchmark-specific logic in adapters outside the shared kernel.

Under a controlled-reference comparison sharing one evaluation harness with a reproduced MemBrain baseline (GPT-4.1-mini backbone), MemStack reaches 93.59% on LoCoMo, 57.24% on PersonaMem-V2, and 87.47% on LongMemEval. The LoCoMo margin over MemBrain (+0.36pp) reflects parity-level competitiveness, while LongMemEval and PersonaMem-V2 show larger relative gains. Optional procedural hooks (user-specific trigger–procedure contracts) remain design-only and are not benchmarked.

Expansion: MemSense and MemFuse

The Evidence Admission layer converts non-dialogue observations into provenance-preserving payloads. MemSense grounds images as identity-preserving objects via an Image Knowledge Base (IKB) with stable IDs, conversation-specific names, categories, and related L0 fact links; an intent router prefers structured IKB resolution over probabilistic VLM perception for visual recall, search/enumeration, and text-to-look naming. On Mem-Gallery (1,711 QA pairs), MemSense achieves 89.15% average judge accuracy (88.19% strict binary) with 35.3s average offline latency; image-related errors account for 58% of residual strict errors. Reference baselines under a unified Qwen-2.5-VL-7B backbone score lower (MuRAG 82.29%), but the authors explicitly decline direct numerical comparison because backbones differ, and an IKB ablation under a shared backbone remains future work.

MemFuse fuses asynchronous device events into MemoryPacks using a three-way LLM judge decision (CreatePack / UpdatePack / Standalone) over a dual-layer graph of AtomicEvent and FusedNode nodes with CAUSAL, SEMANTIC, and BELONG edges. On the internal MemFuseBench, MemFuse k=20 scores 35.2% overall checklist coverage, +4.7pp over mem0@20, leading on causal reasoning, information fusion, and perspective difference—but scoring lower than mem0 on conflict arbitration (33.3% vs. 45.3%) and lower than Direct LLM on multi-user synthesis (25.5% vs. 32.3%). These results are preliminary/internal only; the benchmark lacks human upper bounds or inter-annotator agreement, so absolute scores indicate task-specific coverage rather than normalized competence.

Evolution: D2ACCI and E2MEND

The governance plane separates a human-in-the-loop loop (D2ACCI: Diagnostic-Driven AI-Collaborative Iteration) from automated strategy search (E2MEND). D2ACCI requires falsifiable hypotheses, per-question diagnostic traces routed by a Layer-A classifier to the earliest loss stage (ingestion, retrieval, filtering, generation), paired comparisons aligned by question key, and per-category non-regression gates. A representative round attributed 30% of residual LoCoMo failures to ingestion gaps from long-conversation extraction truncation; adding supplement extraction reduced this to 2% and yielded +4.48pp. Case studies document rejected candidates archived as negative priors, including a dedup-threshold regression traced to cosine-similarity over-merging (0.81 similarity merging distinct date-context facts).

E2MEND automates only schema-constrained mutations within a locked framework—extraction prompts, retrieval parameters, lifecycle policies, feature toggles—with a five-gate fail-fast pipeline, reputation-driven Critic review, and best-ever rollback triggered when accuracy falls more than 3pp below the all-time best checkpoint. On LoCoMo, staged E2MEND evolution improves the baseline from 75.58% to 94.74% (+19.16pp) across six accepted strategy families, with largest gains in single-hop (+139 correct), temporal (+69), and multi-hop (+70) questions. Self-correction and broad raw-dialogue fallback were rejected as regressive or unstable. The authors are explicit that these are descriptive full-benchmark results without repeated-run statistics, and not directly comparable to the earlier MemStack anchor run.

Deployment: LiteMem

LiteMem tests whether the audit contract survives on a repository-native substrate: Markdown/YAML files, file-tool retrieval, Git provenance, and progressive disclosure with lazy loading. Ranking combines lexical relevance, decayed importance, recency, and access feedback; idle organization recomputes importance and commits diffs for auditability. In a LoCoMo-aligned transfer setting, LiteMem reaches 90.81%, retaining 90.0% of the service-side improvement over the same no-memory baseline (65.83%) and a 97.0% absolute ratio against full MemStack. Multi-hop relation traversal is the main remaining limitation, and file-count scaling and latency beyond this setting are unvalidated—the authors omit synthetic scaling tables because repository size was not isolated as a variable.

Limitations and Open Questions

The report maintains an unusually disciplined evidence taxonomy (controlled reference, module-level, preliminary/internal, controlled offline benchmark, transfer-feasibility, design-only) and concedes several boundaries plainly. Privacy enforcement, deletion propagation, cross-user leakage, memory poisoning, consent management, and adversarial validation are explicitly deferred to platform-level layers outside module-level evidence. MemFuseBench lacks external validation; the IKB contribution lacks a shared-backbone ablation; E2MEND results lack statistical significance testing; LiteMem scaling is unmeasured. No closed-loop benchmark exists where memory guides downstream actions, and extreme-scale contexts are untested. Joint ablation across modules—needed to establish marginal contribution rather than contract compatibility—is pending. Five open research problems are named: causal memory attribution over multi-granularity stores, propagation-complete forgetting (invalidating dependent summaries and cached retrievals when evidence is revoked), federated memory composition across devices under differing retention policies, diagnostic-driven online evolution without ground-truth labels, and standardized memory contracts for cross-system ablation.

Conclusion

Mi-Memory's central contribution is the lifecycle audit contract itself: typed evidence, stage-local diagnostics, versioned strategy artifacts, and gate/rollback records that make heterogeneous memory modules jointly inspectable. The empirical anchors are role-specific and deliberately non-comparable—controlled-reference parity-plus on three QA benchmarks, module-level multimodal grounding, preliminary cross-device fusion, descriptive +19.16pp governed evolution, and 90.0%-retention substrate transfer. The framework's value lies less in any single number than in demonstrating that continuity, evidence admission, governed change, and deployment transfer can be evaluated as traceable, bounded claims within one contract, while leaving platform-level safety validation and joint end-to-end ablation as clearly stated open work.