MA-EgoQA: Multi-Agent Egocentric QA Benchmark

• The paper introduces MA-EgoQA, a benchmark that challenges multi-agent temporal reasoning and memory aggregation across concurrent egocentric video streams.
• It employs LLM-assisted synthesis, programmatic templating, and rigorous multi-agent filtering to generate high-quality multiple-choice QA from synchronized video data.
• EgoMAS, the training-free baseline, uses event-based shared memory and dynamic agent-specific retrieval to outperform traditional caption and frame concatenation methods.

MultiAgent-EgoQA (MA-EgoQA) is a benchmark and task formulation for question answering over multiple long-horizon egocentric video streams collected from embodied agents operating concurrently in a shared environment (Kim et al., 10 Mar 2026). It formalizes a setting in which a user query must be answered by integrating information distributed across more than two agents and often across multiple timestamps, rather than by inspecting a single video or a single contiguous moment. The benchmark is built on EgoLife, comprising six temporally aligned egocentric streams recorded over seven days in one house, and contains 1,741 curated multiple-choice questions with five options each. The accompanying baseline, EgoMAS, is a training-free retrieval-and-generation pipeline that combines event-based shared memory with agent-wise dynamic retrieval, exposing the difficulty of system-level memory construction and cross-agent temporal reasoning in current models (Kim et al., 10 Mar 2026).

1. Problem setting and formal definition

MA-EgoQA is defined for a collection of embodied agents $A = \{A_1, \ldots, A_N\}$, where each agent $A_i$ continuously records an egocentric video stream $V_i$ for $T$ hours. The complete multi-agent video collection is denoted as $\mathcal{V} = \{V_1, V_2, \ldots, V_N\}$, yielding a total of $N \times T$ hours of video (Kim et al., 10 Mar 2026). In the benchmark instantiation, the streams are temporally aligned because all agents operate concurrently over seven days in the same house.

The task objective is a mapping

$f : (q, \mathcal{V}) \rightarrow y \in Y,$

where $q$ is a user query and $Y$ is a five-option multiple-choice answer space evaluated by top-1 accuracy (Kim et al., 10 Mar 2026). The paper states that “each agent’s information should be stored in a way that is temporally and contextually aligned with others” and that retrieval is difficult because “queries may require referencing different timestamps across multiple agents” (Kim et al., 10 Mar 2026). This framing distinguishes MA-EgoQA from single-agent egocentric QA and from generic long-video QA by requiring both synchronized multi-stream reasoning and system-level memory aggregation.

A central property of the benchmark is that a single question can reference multiple agents and multiple timestamps across the full seven-day span (Kim et al., 10 Mar 2026). This implies that the dominant failure mode is not only long-context compression but also cross-stream attribution: determining which agents matter, which moments matter, and how their observations should be fused.

2. Benchmark composition and data construction

MA-EgoQA is derived from EgoLife, a corpus of six people living in a shared house for seven days, with a total duration of 266 hours across six egocentric streams (Kim et al., 10 Mar 2026). The benchmark itself contains 1,741 curated multiple-choice questions, each with five answer options. The paper presents MA-EgoQA as an evaluation-only benchmark; no train/validation/test splits are reported (Kim et al., 10 Mar 2026).

Question generation combines LLM-assisted synthesis, programmatic templating, automated filtering, cross-model validation, and human verification. For single-span generation in social interaction, task coordination, and theory-of-mind, each agent’s video is segmented into 5-minute windows, dense captions and transcripts are collected for all agents within each window, and GPT-4o is prompted to generate a question, answer, rationale, referenced agents and timestamps, and four wrong options. The generated candidate counts are reported as 33.4k for social interaction, 31.6k for task coordination, and 34.1k for theory-of-mind (Kim et al., 10 Mar 2026).

For multi-span generation in social interaction and task coordination, semantically similar single-span QA pairs are grouped using embeddings $e = E([q; a])$ and cosine similarity $A_i$0, connected components are constructed with threshold $A_i$1, and GPT-5 synthesizes a multi-span question-answer pair; false options are created separately. The reported candidate counts are 15.9k for social interaction and 16.3k for task coordination (Kim et al., 10 Mar 2026).

Temporal reasoning and environmental interaction are created with template-based generation. Temporal reasoning uses captions at 30-second, 10-minute, and 1-hour scales, with two subtypes: Comparison and Concurrency. GPT-5 then generates the five answer options. Environmental interaction enumerates objects such as microwave and oven, collects captions over 6-hour or 1-day intervals, and instantiates templates asking about first use, most frequent user, counts, or last user (Kim et al., 10 Mar 2026).

Filtering and validation are unusually strict. First, zero-shot filtering asks the model three times with zero context; if it guesses the correct answer in more than two trials, the question is discarded as trivial. Second, single-agent filtering removes any question answerable from only one agent’s memory, selected via names in the question or answer. Third, cross-model validation re-checks questions with Gemini-2.5-Flash and Claude-Sonnet-4, discarding items if either model flags invalidity. Finally, four human verifiers reviewed 3,436 candidates with full access to captions, transcripts, and videos, retaining 1,741 high-quality samples; inter-annotator agreement is not reported (Kim et al., 10 Mar 2026).

3. Question taxonomy and reasoning demands

MA-EgoQA organizes questions into five categories, each intended to be uniquely multi-agent rather than reducible to single-stream inspection (Kim et al., 10 Mar 2026).

Category	Core focus	Typical answer form
Social Interaction (SI)	Conversation and affiliative behavior across streams	Names and brief purpose
Task Coordination (TC)	Roles, division of labor, sequencing toward goals	Role/action descriptions
Theory of Mind (ToM)	Beliefs, misunderstandings, perspectives, intentions	Brief belief statement
Temporal Reasoning (TR)	Concurrency and ordering across agents	Action or event ordering
Environmental Interaction (EI)	Object/device usage and environmental state tracking	Name, timestamp, or count

Social Interaction requires grounding who talked, who responded, and how group behaviors unfolded across streams, including multi-span variants that reason over non-contiguous times (Kim et al., 10 Mar 2026). Task Coordination centers on roles, division of labor, sequencing of actions toward goals, and decisions during execution, again with multi-span variants linking separate times (Kim et al., 10 Mar 2026). Theory of Mind targets beliefs, misunderstandings, perspectives, and intentions, requiring inference about what an agent perceived or failed to perceive, and why (Kim et al., 10 Mar 2026).

Temporal Reasoning has two explicit subtypes. Concurrency asks what one agent did while other agents were performing simultaneous actions; Comparison asks for the correct relative ordering of events across agents at different times (Kim et al., 10 Mar 2026). Environmental Interaction tracks object and device usage across agents and environmental states, including first use, most frequent user, total counts, and last user (Kim et al., 10 Mar 2026).

The paper’s representative question types clarify the expected granularity of responses. Social Interaction includes forms such as “Who helped whom find scissors and for what?” Task Coordination includes “How did the group divide tasks to light the charcoal?” Theory of Mind includes “What did X wrongly assume while watching Y and Z filming?” Temporal Reasoning includes both “When A was doing X and B was doing Y, what was C doing?” and event-ordering questions, while Environmental Interaction includes forms such as “Who used the microwave most on DAY3?” and “How many people used the oven on DAY1?” (Kim et al., 10 Mar 2026). A plausible implication is that benchmark success depends on both relational grounding and fine-grained temporal indexing, rather than only coarse semantic retrieval.

4. EgoMAS: shared memory and agent-wise dynamic retrieval

EgoMAS is the benchmark’s proposed baseline model and is explicitly training-free; no cross-entropy loss or end-to-end optimization objective is defined (Kim et al., 10 Mar 2026). Its architecture is organized around two memory stores: a shared system-level memory $A_i$2 and per-agent memories $A_i$3.

The shared memory is event-based and structured with 4W1H fields: When, What, Where, Who, and How (Kim et al., 10 Mar 2026). At every 10-minute interval, each agent provides a caption summary. A centralized manager then integrates all agents’ summaries into a system-level summary by detecting cross-agent “events” and recording them in 4W1H format, thereby aligning perspectives while preserving salient details. In parallel, each per-agent memory contains agent-specific captions and transcripts at comparable granularity (Kim et al., 10 Mar 2026).

The retrieval pipeline proceeds in stages. First, system-level retrieval performs BM25 over $A_i$4 to return the Top-$A_i$5 event memories relevant to the query, with $A_i$6 (Kim et al., 10 Mar 2026). Second, the model synthesizes an agent-specific request set

$A_i$7

where each $A_i$8 is an agent identifier and each $A_i$9 is a sub-query distilled from the retrieved system-level context (Kim et al., 10 Mar 2026). Third, agent-level retrieval applies BM25 over each relevant agent memory $V_i$0, returning Top-$V_i$1 results with $V_i$2, followed by threshold filtering with $V_i$3 (Kim et al., 10 Mar 2026). Finally, the backbone LLM or VLM receives the original query, the retrieved shared-memory events, and the filtered agent-specific snippets, and selects a final answer option.

The module-level description in the paper identifies six components: per-agent summarization, centralized event extraction, system-level retrieval, agent-wise query synthesis, agent retrieval, and answer generation (Kim et al., 10 Mar 2026). The available backbones include Gemini-2.5-Flash, Qwen3VL-8B-Thinking, Qwen3VL-8B-Instruct, and Qwen2.5VL-7B-Instruct (Kim et al., 10 Mar 2026).

The rationale for EgoMAS is primarily computational and representational. Instead of concatenating all captions or frames from all agents, it compresses long streams into structured summaries and performs targeted retrieval, thereby improving token efficiency and inference latency (Kim et al., 10 Mar 2026). The paper contrasts this with caption-concatenation and frame-concatenation baselines, which are limited by context length and by the inclusion of irrelevant content, and with generic retrieval pipelines that lack structured system-level fusion across agents (Kim et al., 10 Mar 2026).

5. Baselines, evaluation protocol, and empirical findings

The evaluation uses top-1 accuracy overall and per category, with latency analysis reported separately (Kim et al., 10 Mar 2026). Baselines are grouped into caption concatenation, frame concatenation, and retrieval-augmented generation.

Caption-concatenation baselines chronologically concatenate captions from all agents and feed them to LLMs such as Gemini-2.5-Flash, GPT-5, Llama-3.1-Nemotron-8B, Qwen2.5-7B-Instruct-1M, Qwen3-30B, and gpt-oss-120B/20B (Kim et al., 10 Mar 2026). Frame-concatenation baselines concatenate frames across agents and sample within model limits, including VideoChat-Flash, VideoXL-2, and Qwen2.5-VL-7B (Kim et al., 10 Mar 2026). RAG baselines include BM25, DPR with Qwen3-VL-8B-Instruct, VideoRAG with an InternVideo2 encoder, EgoRAG, Ego-R1-Agent-3B, and WorldMM-8B (Kim et al., 10 Mar 2026).

The experimental setup fixes concrete retrieval and context parameters. Caption-concat uses 10-minute windows for Gemini-2.5-Flash, Llama-3.1-Nemotron-8B, and Qwen2.5-7B-Instruct-1M, and 1-hour windows otherwise, with truncation as needed. Frame-concat uses uniform sampling, with limits including 10k frames for VideoChat-Flash, 4k for VideoXL-2, and 1.9k for Qwen2.5VL-7B. RAG baselines retrieve five 30-second clips and four frames per clip, with Qwen3-VL-8B-Instruct as answer generator except for Ego-R1 (Kim et al., 10 Mar 2026).

Method	Overall accuracy	Notes
Gemini-2.5-Flash caption concat	36.93%	1M-token context
GPT-5 caption concat	34.81%	272k context
BM25 RAG	36.01%	Strongest generic RAG baseline
EgoMAS + Gemini-2.5-Flash	41.41%	+4.48% over Gemini caption-concat
EgoMAS + Qwen3VL-8B-Thinking	40.26%	Best TR in table
Oracle + Gemini-2.5-Flash	83.80%	Uses generation context

The main result is that current approaches do not handle multiple egocentric streams effectively, and even very large-context models struggle without retrieval (Kim et al., 10 Mar 2026). Among caption-concat baselines, Gemini-2.5-Flash reaches 36.93% average accuracy, with category scores of 41.22 on SI, 36.36 on TC, 24.26 on ToM, 46.59 on TR, and 33.98 on EI. GPT-5 reaches 34.81% average. Open models in this family typically achieve 21–26% average accuracy (Kim et al., 10 Mar 2026).

Frame-concat baselines perform substantially worse: VideoChat-Flash scores 23.06%, VideoXL-2 scores 20.39%, and Qwen2.5-VL-7B scores 25.22% (Kim et al., 10 Mar 2026). Among RAG baselines, BM25 is strongest at 36.01% average, with category scores of 44.68 on SI, 37.60 on TC, 30.21 on ToM, 33.45 on TR, and 30.64 on EI. WorldMM-8B reaches 27.63%, while EgoRAG, Ego-R1, VideoRAG, and DPR lie in the 19.6–26.2% range (Kim et al., 10 Mar 2026).

EgoMAS improves on these results. With Gemini-2.5-Flash as backbone, it reaches 41.41% average accuracy, with 41.49 on SI, 41.32 on TC, 33.62 on ToM, 39.37 on TR, and 48.19 on EI, representing a +4.48% improvement over Gemini caption-concat under the same backbone (Kim et al., 10 Mar 2026). EgoMAS with Qwen3VL-8B-Thinking reaches 40.26% average and obtains the best TR score in the table at 47.39. EgoMAS with Qwen3VL-8B-Instruct reaches 37.68% average and achieves the best SI score at 43.09. The Oracle setting, which uses the same context employed to generate questions, reaches 83.80% with Gemini-2.5-Flash and 73.98% with Qwen3VL-8B-Instruct (Kim et al., 10 Mar 2026). This suggests that the dominant bottleneck is retrieval and aggregation rather than only answer decoding.

6. Diagnostic analyses, relation to prior work, and limitations

Several analyses characterize the benchmark’s difficulty. Multi-span questions in SI and TC are markedly harder than single-span questions: for EgoMAS with Qwen3-VL-8B-Instruct, SI drops from 46.30% on single-span to 23.08% on multi-span, and TC drops from 43.27% to 23.81% (Kim et al., 10 Mar 2026). ToM is the hardest category overall, with the lowest accuracies across models, including EgoMAS, reflecting the need to infer latent beliefs and intentions rather than directly observed events (Kim et al., 10 Mar 2026). Performance also declines as the number of required agents increases, highlighting the difficulty of multi-agent fusion; conversely, using more available agents improves EgoMAS accuracy from 31.99% to 35.55% (Kim et al., 10 Mar 2026).

The ablations isolate the value of EgoMAS’s two main modules. Under a Qwen2.5VL-7B-Instruct backbone, removing both shared memory and dynamic retrieval yields 27.80%; using shared memory only yields 30.04%; using dynamic retrieval only yields 28.20%; using both yields 35.55%, the best result (Kim et al., 10 Mar 2026). Shared-memory representation also matters: Summary gives 30.67%, Triplet 30.44%, Chunk 25.96%, Graph 31.99%, and the 4W1H structure 35.55% (Kim et al., 10 Mar 2026). For retrievers, DPR reaches 28.67%, Qwen3-Embed-0.6B reaches 33.03%, NV-Embed-v2 7B reaches 37.91%, and BM25 reaches 35.55%; the paper characterizes BM25 as competitive and lightweight relative to heavier dense retrieval (Kim et al., 10 Mar 2026).

Latency analysis indicates that non-retrieval models have high computational overhead, while retrieval-based methods are faster; EgoMAS attains the highest accuracy among retrieval baselines with approximately 1.3 seconds per query on 100-sample averages (Kim et al., 10 Mar 2026). Visual modality helps in some categories—improving SI, TR, and EI for EgoMAS—but may distract in TC and ToM if not selectively filtered (Kim et al., 10 Mar 2026). A plausible implication is that modality selection, not merely multimodal inclusion, is a principal systems challenge for long-horizon multi-agent QA.

Relative to earlier work, the paper positions MA-EgoQA against egocentric QA and long-video understanding datasets such as EgoSchema, EgoThink, EgoToM, EgoExoLearn, and EgoLifeQA, as well as multi-agent embodied systems such as CoELA, Co-NavGPT, ACE, and PARTNR (Kim et al., 10 Mar 2026). MA-EgoQA differs by requiring reasoning over multiple temporally aligned ego streams across seven days, with explicit concurrency and cross-stream alignment. The paper states that EgoLifeQA is single-agent and that MA-EgoQA is the first benchmark to evaluate QA on multiple, temporally aligned ego streams spanning seven days (Kim et al., 10 Mar 2026).

The reported limitations are concrete. The dataset is built only on EgoLife, involving six people, one shared house, and seven days, so broader environments and agent types are needed to test generalization (Kim et al., 10 Mar 2026). Egocentric video also raises privacy concerns because it contains personally identifiable information and private domestic scenes; usage should comply with the source dataset’s privacy policies, consent conditions, and licensing (Kim et al., 10 Mar 2026). The paper further identifies scaling challenges in large token contexts, the inefficiency of naive concatenation, and the importance of retrieval quality and modality selection. Future advances suggested by the authors include stronger or hybrid retrievers, improved shared-memory aggregation beyond 4W1H, cross-agent reasoning protocols, and adaptive multimodality (Kim et al., 10 Mar 2026).