LightMem-Ego: Lightweight Egocentric Multimodal Memory

Updated 14 July 2026

LightMem-Ego is a lightweight system that captures egocentric visual and audio streams and organizes them into current, short-term, and long-term memory.
It employs efficient temporal alignment, event segmentation, and multimodal retrieval routing to support tasks like object finding, conversation recall, and life summarization.
The design integrates on-device sampling with backend processing to optimize storage, latency, and retrieval accuracy while addressing privacy concerns.

Searching arXiv for the named system and closely related memory-based egocentric video work. arxiv_search("LightMem-Ego") arxiv_search(query="LightMem-Ego", max_results=5) LightMem-Ego is a lightweight streaming multimodal memory system for everyday-life assistance that continuously captures egocentric visual and audio streams, aligns them on a shared timeline, and organizes them into a hierarchical memory consisting of current, short-term, and long-term memory. Given a user query, it dynamically routes retrieval to the appropriate memory level and generates answers grounded in multimodal evidence. The demonstrated system is deployable on smartphones and AI glasses, and supports object finding, conversation recall, life summarization, routine discovery, and personalized assistance (Chen et al., 13 Jul 2026).

1. Problem setting and design objectives

LightMem-Ego is motivated by the need for persistent, long-term multimodal memory on mobile phones and AI glasses. The motivating queries are explicitly everyday and temporally extended: “What did I do this afternoon?”, “Where did I leave my keys?”, and “What did the doctor say?”. In the formulation of the system, such queries require assistants to accumulate, organize, and retrieve user-specific experiences over days to weeks, beyond a single context window. At the same time, the target deployment setting imposes tight compute, memory, and battery budgets; continuous streaming imposes bandwidth limits; latency must be near-interactive for short-horizon questions; and storage must be compact and queryable (Chen et al., 13 Jul 2026).

The system description identifies five core challenges. The first is continuous streaming, namely transforming raw frames and audio chunks into coherent, temporally grounded events without semantic parsing of all frames. The second is alignment, specifically synchronizing visual frames and audio on a shared timeline and backfilling transcripts as ASR finishes. The third is organization, converting micro-events into episodic memory and semantic facts while keeping indices light. The fourth is retrieval, routing queries to the correct temporal scope—current, short-term, or long-term—and returning multimodal evidence efficiently. The fifth is grounding, generating answers that cite representative frames, timestamps, and transcript snippets for faithfulness (Chen et al., 13 Jul 2026).

A common misconception is to equate the label “lightweight” with fully on-device multimodal inference. In LightMem-Ego, the client handles sampling, compression, timestamping, and session management, whereas the backend performs ASR backfilling, event refinement, indexing, fusion, semantic extraction, retrieval, and LLM generation. The lightweight character of the prototype therefore comes primarily from hierarchical organization, streaming optimization, and offloading of heavy inference, rather than from an entirely local VLM/LLM stack (Chen et al., 13 Jul 2026).

2. Stream representation and hierarchical memory

The unified streaming interface builds a temporally ordered sequence

$X = \{x_t\}_{t=1}^{T}, \qquad x_t = (v_t, a_t, m_t),$

where $v_t$ is the egocentric visual stream, $a_t$ is the audio stream, and $m_t$ denotes optional metadata such as coarse location, screen activity, and lightweight sensor cues. Relative session time is defined as

$\tau = t - t_0,$

and is used for downstream event building and retrieval (Chen et al., 13 Jul 2026).

The memory hierarchy is written as

$M = \{M_{cur}, M_{st}, M_{lt}\}.$

Its three levels differ in temporal scope, data structure, and representational purpose.

Memory level	Data structure	Stores
$M_{cur}$	Fixed-size ring buffers, active segment pointer	Raw observations, quick visual tags, in-progress transcript snippets
$M_{st}$	Bounded deque of segment records	Micro-events with fused embeddings and light metadata
$M_{lt}$	Document store, vector indices, relational links	Episodic memory and semantic memory

Current memory, $M_{cur}$ , is a rolling buffer over the most recent multimodal observations and active event state. Its capacity is the last $v_t$ 0 seconds, or $v_t$ 1 frames/chunks. Short-term memory, $v_t$ 2, is a queue or index of recent event segments, each with embeddings $v_t$ 3, timestamps, representative frames, provisional or refined descriptions, and transcript snippets. Its capacity is the last $v_t$ 4 segments or the last $v_t$ 5 minutes. Long-term memory, $v_t$ 6, is divided into an episodic memory $v_t$ 7 and a semantic memory $v_t$ 8. $v_t$ 9 stores event-centered records—segment clusters into episodes with canonical summaries, key frames, timestamps, and fused embeddings—while $a_t$ 0 stores higher-level regularities such as routines, preferences, and relationships, represented as lightweight facts with confidence and provenance (Chen et al., 13 Jul 2026).

This organization is central to the retrieval model. LightMem-Ego does not treat all stored evidence as a flat archive. Instead, it explicitly distinguishes current sensory context, recent micro-events, and consolidated episodic or semantic knowledge, so that temporal scope becomes a first-class retrieval variable rather than a by-product of embedding similarity alone (Chen et al., 13 Jul 2026).

3. Alignment, event construction, and retrieval routing

Temporal alignment is formulated through modality-specific mappings from device time to session time:

$a_t$ 1

Over a window $a_t$ 2, the synchronization objective is

$a_t$ 3

and the parameters $a_t$ 4 are adjusted periodically to minimize $a_t$ 5 (Chen et al., 13 Jul 2026).

The stream is incrementally partitioned into short event segments

$a_t$ 6

each a contiguous interval on the shared timeline. Lightweight detectors use temporal continuity and visual change, optionally audio activity, to open and close segments. Each segment stores start and end time, representative frames, a provisional visual description, and pending or aligned transcript snippets. For a closed segment $a_t$ 7, visual and audio-text descriptors are embedded as $a_t$ 8 and $a_t$ 9, and fused into an event-level embedding through either a linear form

$m_t$ 0

or an attention-based form

$m_t$ 1

Near-duplicate detection uses cosine similarity,

$m_t$ 2

with merge or duplicate marking when $m_t$ 3 and the segments overlap in time (Chen et al., 13 Jul 2026).

Promotion from short-term to long-term memory is governed by a segment stability score

$m_t$ 4

with promotion when $m_t$ 5. Forgetting and demotion are modulated by

$m_t$ 6

and stale items are demoted from $m_t$ 7 once $m_t$ 8 falls below threshold or when capacity exceeds $m_t$ 9 (Chen et al., 13 Jul 2026).

At query time, LightMem-Ego parses temporal scope, entities, and intent, embeds the query as $\tau = t - t_0,$ 0, and scores each memory level by

$\tau = t - t_0,$ 1

Routing is then

$\tau = t - t_0,$ 2

The system also applies hard priors: “now” or “this moment” forces $\tau = t - t_0,$ 3; “earlier today” or “this afternoon” prefers $\tau = t - t_0,$ 4; and “usually” or “last month” prefers $\tau = t - t_0,$ 5. After routing, retrieval performs cross-modal search over event embeddings and text indices, reranks with evidence completeness,

$\tau = t - t_0,$ 6

and constructs an evidence view containing timestamps, episode or segment IDs, representative frames, transcript snippets, and brief summaries. The answer generator then produces

$\tau = t - t_0,$ 7

with a structured output template that includes answer, evidence, and transcript quotation (Chen et al., 13 Jul 2026).

4. Tasks, deployment profile, and empirical evaluation

The prototype is organized as a client-backend system. The client performs sampling, compression, timestamping, and session management; the backend handles ASR backfilling, event refinement, indexing, fusion, semantic extraction, retrieval, and LLM generation. The paper emphasizes rolling buffers for $\tau = t - t_0,$ 8, bounded deques for $\tau = t - t_0,$ 9, asynchronous consolidation to $M = \{M_{cur}, M_{st}, M_{lt}\}.$ 0, precomputed embeddings, and lightweight indices keyed by time or event as the main streaming optimizations (Chen et al., 13 Jul 2026).

The demonstration supports five concrete tasks. Object finding routes to short-term or long-term memory, finds the last segment where the queried object appears in visual or textual evidence, and returns location, time, and representative frame. Conversation recall retrieves recent conversational segments, aligns the transcript, cites the speaker and exact quote, and provides a timestamp. Life summarization aggregates segments over a period such as 12–18h and summarizes activities with times and key frames. Routine discovery searches long-term semantic memory for recurring patterns such as “coffee → email → standup”. Personalized assistance follows from the same retrieval and grounding pathway, but with user-specific evidence and long-term regularities (Chen et al., 13 Jul 2026).

Prototype latencies were reported separately for short-term and long-term question answering. For short-term memory QA, retrieval is 13–15 ms (P50–P90, phone) and 14–29 ms (glasses-style), while answer generation is 5.77–10.38 s (phone) and 6.10–9.79 s (glasses-style). End-to-end latency is 5.86–10.95 s on phone and 7.01–9.96 s on glasses-style hardware. For long-term memory QA, retrieval is 4.09–15.39 s (phone) and 10.39–28.93 s (glasses-style), answer generation is 9.00–22.40 s (phone) and 9.25–22.62 s (glasses-style), and end-to-end latency is 14.87–35.15 s (phone) and 19.96–42.70 s (glasses-style) (Chen et al., 13 Jul 2026).

The reported evaluation uses three scenarios—object finding, conversation recall, and life summarization—with manually annotated gold evidence.

Scenario	Retrieval	QA accuracy
Object finding	R@1=22.2, R@3=66.7, R@5=77.8, MRR=0.454	44.4 / 55.6
Conversation recall	R@1=44.4, R@3=55.6, R@5=55.6, MRR=0.481	33.3 / 33.3
Life summarization	R@1=88.9, R@3=100.0, R@5=100.0, MRR=0.944	77.8 / 77.8

Overall retrieval is R@1=51.9, R@3=74.1, R@5=77.8, MRR=0.627, and overall QA accuracy is 51.9 / 55.6 for LLM-judge and human evaluation, respectively. The paper notes that formal baseline numbers are not detailed, and that the demo-level evaluation does not report formal ablations. This suggests that the reported system should be read primarily as a deployed research prototype rather than as a fully benchmarked replacement for mature long-horizon multimodal retrieval systems (Chen et al., 13 Jul 2026).

5. Relation to AMEGO and adjacent memory systems

LightMem-Ego is explicitly connected to earlier work on egocentric memory. In AMEGO, the underlying representation is an active memory

$M = \{M_{cur}, M_{st}, M_{lt}\}.$ 1

where $M = \{M_{cur}, M_{st}, M_{lt}\}.$ 2 stores hand–object interaction tracklets and $M = \{M_{cur}, M_{st}, M_{lt}\}.$ 3 stores location segments. AMEGO constructs this memory in a single pass, captures key locations (“hot-spots”) and hand–object interactions, is semantic-free, and enables multiple queries without reprocessing the entire visual content. Query answering is then reduced to feature matching and interval operations over stored instances and time intervals, supporting sequencing, concurrency, and temporal grounding (Goletto et al., 2024).

The AMEGO paper also introduces the Active Memories Benchmark, comprising 20.5K multiple-choice, vision-only queries on 100 EPIC-KITCHENS videos with average length ~14 minutes. Its query templates cover object sequences, before/after reasoning, object–object and object–location concurrency, and temporal grounding. On this benchmark, AMEGO-S reaches 33.8% total accuracy and AMEGO-L reaches 36.3%, with AMEGO-L averaging +12.7% over baselines. In the same paper, the “Toward LightMem-Ego” discussion states that a lightweight configuration mirrors AMEGO-S with conservative start/end filters and slightly stricter clustering, preserving one-pass memory construction and multi-query answering through feature matching and interval operations (Goletto et al., 2024).

A plausible implication is that the 2026 LightMem-Ego system extends the memory-first philosophy of AMEGO from visual-only, semantic-free active memory to streaming multimodal memory with audio, ASR transcripts, hierarchical consolidation, and query routing. The two systems are not identical: AMEGO is centered on instance-centric visual memory for very-long egocentric videos, whereas LightMem-Ego is centered on multimodal everyday-life assistance with current, short-term, and long-term memory (Goletto et al., 2024).

The name also appears in neighboring memory literature in a more architectural or interpretive sense. The LightMem paper organizes agent memory into STM, MTM, and LTM, with ego-centricity realized by the per-user MTM store $M = \{M_{cur}, M_{st}, M_{lt}\}.$ 4 rather than by a module named Ego; it reports about +2.5 average F1 on LoCoMo and low median latency of 83 ms retrieval and 581 ms end-to-end (Zhang et al., 9 Apr 2026). EgoMem, by contrast, is a lifelong memory agent for full-duplex omnimodal models that operates with retrieval, dialog, and memory management as three asynchronous processes over raw audiovisual streams, and its accompanying technical note explicitly frames “LightMem-Ego” as a lightweight egocentric lifelong memory agent inspired by EgoMem (Yao et al., 15 Sep 2025). These formulations show that “LightMem-Ego” functions both as the title of a concrete 2026 multimodal memory system and as a broader design direction for lightweight, ego-centric, memory-first assistants.

6. Limitations, privacy, and open research directions

The current LightMem-Ego system is presented as a research prototype rather than a completed privacy-preserving personal memory substrate. The paper states directly that it does not yet implement a full privacy-preserving pipeline. This is significant because the captured streams contain sensitive content: faces, conversations, documents, screens, locations, and routines. Persistent multimodal memory enlarges privacy exposure both for raw data and for derived representations such as summaries, embeddings, and semantic facts (Chen et al., 13 Jul 2026).

The intended protections are listed, but not fully realized: on-device selective capture, sensitive-content filtering or redaction, encrypted storage, user-controlled retention, edit and delete functions, bystander consent cues, and privacy-aware consolidation to avoid promoting sensitive items to long-term memory. These design points indicate that storage policy and memory promotion are privacy-critical, not merely systems-level optimization choices (Chen et al., 13 Jul 2026).

Technically, several limitations remain. Accuracy and latency depend on upstream ASR, VLM, and LLM APIs, with rate limits and runtime variation. Alignment and transcription errors can propagate into memory and affect answer faithfulness. Continuous segmentation, summarization, embedding, indexing, and storage introduce overhead. The memory lifecycle is preliminary and lacks principled policies for revising, merging, forgetting, or promotion. Edge cases include noisy audio, occlusions, domain shifts, long-horizon causal reasoning gaps, and personalization trade-offs between precision and privacy (Chen et al., 13 Jul 2026).

The stated future directions are robust alignment, adaptive promotion and forgetting policies, on-device efficient models, richer semantic memory induction, formal ablations, and larger-scale benchmarks such as LifeEval and egocentric retrieval or VQA datasets. This suggests that the present contribution lies less in a final memory architecture than in a deployable systems template: unified multimodal streaming, hierarchical memory, query routing by temporal scope, and evidence-grounded answer generation (Chen et al., 13 Jul 2026).