EgoMAS: Multi-Agent Shared Memory Baseline

Updated 4 July 2026

EgoMAS is a training-free multi-agent retrieval-and-reasoning baseline that integrates a shared event memory to address cross-agent egocentric video QA.
The system employs a two-stage retrieval process: first, fusing periodic 4W1H event summaries and then executing agent-specific dynamic retrieval to refine evidence.
Empirical results on MA-EgoQA show EgoMAS outperforms simple concatenation baselines, highlighting the significance of structured memory and specialized retrievers.

EgoMAS is a training-free multi-agent retrieval-and-reasoning baseline introduced for MultiAgent-EgoQA (MA-EgoQA), a benchmark for question answering over multiple long-horizon egocentric video streams collected from embodied agents. Its defining idea is to avoid naively concatenating all multi-agent history and instead build a shared event memory across agents, then perform query-conditioned, agent-specific retrieval before answer generation. In this formulation, EgoMAS is neither a monolithic video-LLM nor a conventional single-agent egocentric QA system; it is a centralized memory-and-retrieval pipeline for system-level understanding across multiple first-person timelines (Kim et al., 10 Mar 2026).

1. Problem setting and scope

EgoMAS is defined within the problem of Multi-Agent Egocentric Video Question Answering, where there are $N$ embodied agents, each recording an egocentric stream for $T$ hours, yielding a collection

$\mathcal{V} = \{V_1, V_2, \dots, V_N\}.$

The central difficulty is that the query is not answerable from one wearer’s memory alone; it requires information from more than two agents, often also from different times. This imposes two coupled requirements: global understanding across agents and query-conditioned retrieval across time and agents (Kim et al., 10 Mar 2026).

MA-EgoQA is built on EgoLife, a dataset of 266 hours of egocentric video captured by 6 people living in a shared house for 7 days. It contributes 1,741 5-way multiple-choice QA pairs spanning five categories: Social Interaction (SI), Task Coordination (TC), Theory of Mind (ToM), Temporal Reasoning (TR), and Environmental Interaction (EI). The benchmark is intentionally filtered so that questions requiring only one person’s memory are removed, and many questions require evidence from multiple agents and multiple timestamps. In that setting, EgoMAS serves as a baseline specifically designed for multi-stream egocentric memory search rather than for generic long-context prompting (Kim et al., 10 Mar 2026).

This benchmark context is crucial for understanding what EgoMAS is and is not. It is not a general egocentric assistant framework, and it is not primarily a perception model. It is a structured baseline for answering multi-agent egocentric questions under severe context-length constraints.

2. Core architecture

EgoMAS is described as a centralized multi-agent system with two principal components: event-based shared memory and agent-wise dynamic retrieval. Each embodied agent periodically emits summaries of its observations, and a centralized manager fuses them into a system-level memory. At question time, EgoMAS first retrieves relevant entries from this shared memory, then generates agent-specific retrieval requests, and finally answers using the original query plus the retrieved evidence (Kim et al., 10 Mar 2026).

The architecture is intentionally simple. Every 10-minute interval, each agent produces a caption summarizing its observations during that period. A centralized manager combines the six agents’ captions into a system-level summary and records key events using explicit 4W1H fields:

When
What
Where
Who
How

This 4W1H representation is the preferred memory structure in the paper because it temporally aligns viewpoints, fuses fragmented observations, and preserves event attributes useful for downstream retrieval. The system then performs a two-stage search. First, it retrieves from the shared memory $\mathcal{M}_{\text{shared}}$ with BM25: $\mathcal{R}_{\text{sys}}(q) = \text{Top-}n \; \{ (m, s(m,q)) \mid m \in \mathcal{M}_{\text{shared}} \}.$ Second, from the retrieved shared-memory context, it generates a set of agent-specific retrieval instructions,

$\mathcal{Q}_{\text{agent}} = \{ (a_j, q_j) \}_{j=1}^J,$

where each pair specifies an agent $a_j$ and a tailored sub-query $q_j$ . For each selected agent,

$\mathcal{R}_{a_j}(q_j) = \text{Top-}k \; \{ (m, s(m,q_j)) \mid m \in \mathcal{M}_{a_j} \},$

followed by threshold filtering,

$\widetilde{\mathcal{R}_{a_j}(q_j)} = \{ (m, s(m,q_j)) \in \mathcal{R}_{a_j}(q_j) \mid s(m,q_j) \geq \tau \},$

and aggregation,

$T$ 0

The final answer is generated as

$T$ 1

This design makes EgoMAS a hierarchical retrieval system: coarse event localization at the shared-memory level, then fine-grained evidence collection at the agent level. The paper explicitly characterizes it as a training-free baseline, not a learned end-to-end architecture in the conventional sense (Kim et al., 10 Mar 2026).

3. Memory model and retrieval semantics

The distinctive feature of EgoMAS is that the shared memory is not just a concatenation of captions. The centralized manager fuses multi-agent captions into event-based entries with 4W1H fields, and this structured memory outperforms alternative memory forms tested in the paper. In the reported ablation, Summary achieved 30.67, Triplet 30.44, Chunk 25.96, Graph 31.99, and 4W1H 35.55, showing that event structure matters materially for retrieval quality (Kim et al., 10 Mar 2026).

The second distinctive feature is agent-wise dynamic retrieval. The query is not broadcast unchanged to all memories. Instead, EgoMAS uses the retrieved shared-memory context to produce different sub-queries for different agents. This matters because one agent may contain evidence about who initiated an interaction, another may contain what followed, and a third may contain the relevant object or environmental state. The system therefore retrieves evidence in a perspective-specific way rather than treating all first-person streams as a flat corpus.

This design also makes EgoMAS fundamentally different from standard single-agent egocentric QA systems. In single-stream settings, retrieval typically searches one long timeline. EgoMAS instead assumes fragmented, overlapping, partial observations distributed across several agents and uses the shared event memory to decide which viewpoints deserve detailed inspection. A plausible implication is that the shared-memory manager functions as a coarse system-level alignment mechanism, while the dynamic retrieval stage handles perspective specialization.

The default retriever in both stages is BM25. The reported hyperparameters are $T$ 2 for shared-memory retrieval, $T$ 3 for per-agent retrieval, and $T$ 4 for filtering. The paper also evaluates alternative retrievers: DPR scored 28.67, Qwen3-Embed-0.6B 33.03, NV-Embed-v2 7B 37.91, and BM25 35.55. This indicates that BM25 is not the strongest retriever available, but it remains competitive and practical, while stronger dense retrieval appears computationally heavier (Kim et al., 10 Mar 2026).

4. Empirical performance and diagnostic findings

The central quantitative result is that EgoMAS outperforms direct long-context baselines and generic retrieval baselines on MA-EgoQA. The strongest reported EgoMAS variant, EgoMAS (Gemini-2.5-Flash), reaches 41.41% average accuracy. For comparison, Gemini-2.5-Flash with all-caption concatenation reaches 36.93%, BM25 reaches 36.01%, and WorldMM-8B reaches 27.63%. Other EgoMAS variants include 40.26 for Qwen3VL-8B-Thinking, 37.68 for Qwen3VL-8B-Instruct, and 35.55 for Qwen2.5VL-7B-Instruct. The reported oracle remains far higher, at 83.80 with Gemini-2.5-Flash, so the benchmark is far from saturated (Kim et al., 10 Mar 2026).

System	Average accuracy
EgoMAS (Gemini-2.5-Flash)	41.41
EgoMAS (Qwen3VL-8B-Thinking)	40.26
Gemini-2.5-Flash (all-caption)	36.93
BM25	36.01
WorldMM-8B	27.63
Oracle (Gemini-2.5-Flash)	83.80

The ablations isolate the value of EgoMAS’s two defining components. With no shared memory and no dynamic retrieval, accuracy is 27.80. With dynamic retrieval only, it is 28.20. With shared memory only, it rises to 30.04. With both shared memory and dynamic retrieval, it reaches 35.55. This shows that both components matter, and that shared memory contributes more than dynamic retrieval alone.

Category-level analyses reveal that multi-agent reasoning remains difficult, especially for Theory of Mind and multi-span questions. The paper notes that ToM is the hardest category and that performance drops sharply when questions require connecting temporally separated events. It also reports that as the number of required agents increases, performance declines, and that single-agent-only variants perform substantially worse. These findings support the benchmark’s claim that MA-EgoQA genuinely tests cross-agent reasoning rather than single-stream recall.

Efficiency is also a reported advantage. Retrieval-based methods are much faster than non-retrieval baselines, and EgoMAS (Qwen2.5VL-7B-Instruct) achieves the highest accuracy among retrieval-based methods while requiring only 1.3 seconds per query. The paper presents this as evidence that hierarchical retrieval is not only more accurate than full-context concatenation but also more computationally tractable (Kim et al., 10 Mar 2026).

5. Limitations, misconceptions, and open problems

A common misconception is that EgoMAS is a full, learned, end-to-end multi-agent architecture. In published form, it is not. It is explicitly presented as a training-free baseline built from shared memory, lexical retrieval, agent-specific sub-query generation, and a responder model. Its value lies in making multi-agent egocentric QA tractable and diagnosable, not in claiming to solve the problem outright (Kim et al., 10 Mar 2026).

A second misconception is that simply increasing context length should match or exceed EgoMAS. The reported results argue otherwise. Naive all-caption concatenation suffers from too much irrelevant information, and all-frame concatenation performs even worse because it lacks transcript information and includes many irrelevant frames. The paper’s qualitative analysis attributes part of EgoMAS’s advantage to its ability to localize relevant events before invoking perspective-specific retrieval.

The method also has explicit weaknesses. The gap to oracle is about 42.4 percentage points, which the paper identifies as evidence that retrieval quality and evidence integration remain major bottlenecks. Multi-span reasoning remains hard. Many-agent fusion remains weak. The appendix reports that adding video frames helps on SI, TR, and EI, but slightly hurts on TC and ToM, suggesting that irrelevant frames can distract the responder if they are not selected carefully. This suggests that modality selection itself is an unresolved systems problem, not merely a backbone issue.

The retrieval ablation implies another limitation. Although BM25 is the default retriever, NV-Embed-v2 7B reaches 37.91, above BM25’s 35.55. A plausible implication is that the present EgoMAS architecture is partly retriever-limited rather than concept-limited. The paper explicitly identifies retrieval quality as a promising future direction and suggests hybrid lexical-plus-embedding retrieval and better decisions about when to use video frames (Kim et al., 10 Mar 2026).

The name EgoMAS has a precise published meaning in MA-EgoQA: a shared-memory baseline for multi-agent egocentric video QA. However, the term also resonates with two adjacent research lines that do not use the label as an official system name. One is ego-state-based multi-agent systems grounded in Transactional Analysis, and the other is egocentric assistant evaluation under streaming memory constraints. These are related conceptually but distinct in scope and formalization (Zamojska et al., 28 Jul 2025, Zamojska et al., 18 Dec 2025, Yan et al., 28 Oct 2025).

In the Transactional Analysis line, Trans-ACT implements each agent as an internal decomposition into Parent, Adult, and Child ego states, each modeled as a separate ReAct agent with distinct memory access, followed by a decision-making agent whose selection is influenced by the agent’s life script. A later paper formalizes a similar response-selection process as

$T$ 5

where the final response $T$ 6 is selected from ego-state proposals using life script $T$ 7 and context $T$ 8. Those papers present an ego-state-based multi-agent system, but they do not explicitly introduce a system named EgoMAS. This suggests a broader, non-canonical sense in which “EgoMAS” can denote multi-agent systems structured around internal ego states rather than around multi-agent egocentric video memory (Zamojska et al., 28 Jul 2025, Zamojska et al., 18 Dec 2025).

By contrast, TeleEgo is not a model architecture at all but a benchmark for streaming egocentric AI assistants with synchronized video, audio, and text, evaluated by Real-Time Accuracy and Memory Persistence Time. It is relevant because it measures capabilities—streaming inference, long-horizon memory, temporal grounding, cross-memory reasoning—that a practical egocentric assistant or an EgoMAS-like system might need. But TeleEgo is a benchmark and evaluation protocol, not a multi-agent shared-memory retrieval baseline (Yan et al., 28 Oct 2025).

These distinctions matter. In the current literature, EgoMAS most concretely denotes the MA-EgoQA baseline. Ego-state MAS and streaming egocentric assistant benchmarks are conceptual neighbors, not terminological equivalents.