---
title: 'EgoTeam: Multi-Robot Egocentric Spatial Reasoning'
url: https://www.emergentmind.com/topics/egoteam
type: topic
---

# EgoTeam: Multi-Robot Egocentric Spatial Reasoning

Searching arXiv for EgoTeam and closely related egocentric reasoning benchmarks to ground the article in current papers.
EgoTeam is a multi-robot egocentric question-answering dataset introduced as the core data resource of the CoopSR benchmark for multi-robot cooperative dynamic spatial reasoning. It is designed to evaluate whether models, especially multimodal large language models, can integrate synchronized egocentric videos from multiple moving robots in order to answer spatial, temporal, visibility, and coordination questions that require team-level reasoning rather than single-view recognition. The dataset is paired with simulated environments in Habitat and iGibson and a real-world test set collected with quadruped robots, and it is presented together with the SP-CoR framework in “Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models” [2605.18431].

## 1. Definition and scope

EgoTeam targets cooperative egocentric spatial reasoning in settings where multiple embodied agents observe the world simultaneously from distinct first-person viewpoints. Its central premise is that the relevant inference problem is not limited to what one robot sees, but extends to what a team collectively perceives, knows, and can coordinate to do. The benchmark therefore evaluates reasoning over synchronized multi-robot videos rather than over isolated clips or disembodied scene descriptions [2605.18431].

The dataset is framed around multi-robot cooperative dynamic spatial reasoning. Questions require integration of evidence across robots and across time, and span spatial, temporal, visibility, and coordination phenomena. This distinguishes EgoTeam from benchmarks centered on a single egocentric stream or on non-embodied multi-view input. A recurring misconception is to treat EgoTeam as a straightforward extension of single-video QA to several cameras; the benchmark definition instead emphasizes team awareness, conflict detection, cross-view object identification, team belief updates, task assignments, and global event reasoning.

A plausible implication is that EgoTeam occupies an intermediate position between embodied perception benchmarks and multi-agent reasoning benchmarks: it uses first-person sensory streams as primary input, but the target capability is cooperative inference at the team level.

## 2. Dataset composition and corpus organization

EgoTeam contains **114,227 QA pairs** spanning **19 question types**, **four difficulty tiers**, and **three team sizes**. Teams contain **2, 3, or 4 robots per scenario**, and clips range from **1 to 6 minutes**. The summary in the benchmark description also characterizes the corpus as **21K+ clips**, with cooperative, multiview, multi-robot dynamics across simulated and real settings [2605.18431].

| Component | Value | Notes |
|---|---:|---|
| QA pairs | 114,227 | 19 question types |
| Difficulty tiers | 4 | T1 to T4 |
| Team sizes | 2, 3, 4 | Variable-sized teams |
| Simulators | Habitat, iGibson | Train/val/test splits provided |
| Real-world test | 2,326 QAs | 13 real scenes, two quadruped robots |
| Clip duration | 1 to 6 minutes | Synchronized egocentric videos |

The simulated portion uses **Habitat** and **iGibson**. Habitat provides **119/12/23 scenes for train/val/test**, while iGibson provides **8/2/4 scenes for train/val/test**. The real-world component consists of **around 2,326 QAs** collected in **13 real scenes** using **two quadruped robots** and motion-capture-based pose annotation.

Each scenario contains synchronized egocentric observations for all robots in the team. The recorded data include exploration trajectories and robot-object interaction events such as object pushing, together with metadata including robot and object poses, egocentric RGB-D and semantic video, segmentation, and scene object relationships at every timestep. This organization makes the dataset suitable for studying both local perception and cross-robot evidence fusion.

## 3. Question taxonomy and difficulty structure

The 19 QA types are organized over **five axes: spatial, exploration, relational, robot action, visibility**. They are also grouped into **four difficulty levels**, denoted **T1** through **T4**, which impose a progressively stronger requirement for cross-view and cross-temporal integration [2605.18431].

**T1: Egocentric Spatial QA** focuses on a single robot and tests spatial awareness, including questions such as where an object is and its direction, distance, or layout relative to the observer.

**T2: Pairwise Relationship Reasoning** covers relations between two entities, including robot-robot and robot-object relations. It includes visibility and occlusion queries such as which robots can see an object, as well as relative position relations such as left/right and front/behind.

**T3: Scene-level Composition & Robot Action Reasoning** requires the integration of multiple views and temporal information. It includes tracking object changes and actions, comparing shared objects across viewpoints, and tracking robot approaches or departures over time.

**T4: Multi-Robot Dynamic Spatial Reasoning** is the most cooperative tier. It includes team awareness, conflict detection, cross-view object identification, team belief updates, task assignments, and global event reasoning.

All questions use a **multiple-choice MC4** format with **4 options, A/B/C/D**, and models answer by selecting one letter. This strict answer space supports controlled benchmarking via accuracy. The tiered structure is intended to separate local egocentric competence from genuinely cooperative reasoning. This suggests that improvements on T1 do not by themselves establish competence on the higher-order coordination demands represented by T4.

## 4. Environments, sensing, and annotation pipeline

The simulated environments are complementary. **Habitat** is described as a high-performance, photorealistic indoor simulation environment with **154 floor plans**, while **iGibson** is an interactive, physics-based indoor household simulation with **14 scenes** supporting diverse robot-object interactions [2605.18431].

Within these environments, teams of 2, 3, or 4 robots perform coverage exploration and collect synchronized egocentric observations. Per-episode data include egocentric RGB-D videos, semantic masks, depth, odometry, and ground-truth poses. Robot-object interaction events are explicitly included to generate varied spatial and temporal scenarios.

The real-world test set uses **two Unitree quadruped robots**. It spans **13 scenes**, each set up with everyday objects, with each scene lasting **1.75h in total**. Human annotators provide metadata and QA pairs, and the annotation process uses intervals ensuring scene consistency with **<30 frames between keyframes**. Manual review by **4 annotators** yields a **high agreement ratio (0.9)**.

A second misconception is that EgoTeam is primarily synthetic and therefore disconnected from deployment conditions. The benchmark explicitly includes a real-world test set and sim-to-real evaluation. At the same time, the authors acknowledge that the real-world set is small compared to simulation, which places a practical limit on the diversity of the real-domain evaluation.

## 5. Formal task definition and evaluation protocol

Formally, for each robot \( r \) in a team of size \( N \), the input is a video stream
\[
\mathcal{V}_r = \{\mathbf{x}_{r,t}\}_{t=1}^{T},
\]
together with a query \( \mathbf{q} \). During training, pose data are available as robot states in \( \mathrm{SE}(3) \); at test time, the model must answer using only the egocentric videos and the query. The prediction task is written as
\[
\mathbf{y} = f_\theta(\mathcal{V}, \{\hat{\mathbf{u}}_{r,t}\}, \mathbf{q}) \text{ during training}
\]
and
\[
\mathbf{y} = f_\theta(\mathcal{V}, \mathbf{q}) \text{ during testing},
\]
where \( \mathcal{V} = \{\mathcal{V}_r\}_{r=1}^{N} \) [2605.18431].

The primary metric is **MC4 Accuracy**, defined as the percentage of correctly answered multiple-choice questions. Evaluation is reported as overall average accuracy and also broken down by **difficulty tier (T1-T4)**, **team size (N = 2, 3, 4)**, and **environment (Habitat vs iGibson)**. Two additional protocols are emphasized:

- **Cross-team-size generalization**: train on \( N=2,3 \), evaluate on \( N=4 \).
- **Sim-to-real transfer**: evaluate on real-world scenes unseen during training.

The benchmark paper evaluates **22 MLLM baselines**, including zero-shot MLLMs, supervised-finetuned backbones, retrieval-augmented generation systems, and keyframe selection methods. The proposed **SP-CoR** framework combines dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation; it outperforms the strongest fine-tuned baseline by **+3.87% on Habitat** and **+7.12% on iGibson**, and shows stronger generalization to unseen team sizes and real-world robot tests [2605.18431].

## 6. Position within egocentric reasoning research, novelty, and limitations

EgoTeam is presented as the **first benchmark for multi-robot cooperative dynamic spatial reasoning**, and the dataset is described as the first to enable **fine-grained cooperative spatial reasoning from multi-robot egocentric videos** [2605.18431]. Its novelty lies in benchmarking questions that explicitly require multi-robot integration, including cross-view and cross-temporal reasoning, rather than only single-agent egocentric understanding.

This positioning becomes clearer when contrasted with adjacent egocentric QA work. For example, EgoToM benchmarks Theory-of-Mind reasoning from egocentric videos using over **1,000 multi-choice QA instances** across **785 unique Ego4D video clips**, with tasks for goal, belief, and future action inference, but it remains centered on a single camera wearer rather than robot-team cooperation [2503.22152]. EgoTeam therefore shifts the emphasis from individual mental-state inference to collective embodied reasoning over synchronized robotic viewpoints.

The benchmark description also argues that most previous datasets, including Ego4D-oriented or single-agent egocentric QA resources, do not provide team-based cooperation or cross-robot dynamics and relations at comparable granularity. This suggests that EgoTeam addresses a gap between single-agent egocentric video understanding and multi-agent embodied intelligence.

Its limitations are stated directly: the **real-world set is small compared to simulation**, and future expansions are needed for greater diversity and scalability. That limitation is important for interpreting reported sim-to-real performance. The benchmark nonetheless provides a structured basis for evaluating whether models can progress from local first-person perception to team-level spatial reasoning, coordination, and evidence integration.

Source: https://www.emergentmind.com/topics/egoteam