MM-EQA Benchmark: Multi-Agent Embodied QA

• The MM-EQA benchmark is a multi-agent, multi-task testbed that emphasizes cooperative exploration and natural-language communication in complex 3D household scenes.
• It employs semantic value maps and conformal prediction to guide decentralized agents in answering questions efficiently while minimizing exploration time.
• Evaluated on Habitat-simulated HM3D environments, MM-EQA provides insights into the effects of scene size, communication latency, and team scalability on task success.

The MM-EQA benchmark is a benchmark for multi-agent multi-task Embodied Question Answering, introduced alongside CommCP to study fully cooperative information gathering in shared 3D household environments. In this setting, multiple robots begin from different initial poses, receive multiple embodied questions, communicate in natural language, and must maximize question-answering success while minimizing exploration time before a common horizon $T_\text{max}$ (Zhang et al., 5 Feb 2026). Unlike canonical single-agent EQA, the benchmark makes communication, coordination, and non-redundant exploration central parts of the task definition.

1. Formal problem setting

CommCP defines an MM-EQA instance as

$\xi:= (E, G_0, T_\text{max}, Q, Y),$

where $E$ is a 3D scene of size $L \times W \times H$, discretized into a voxel map $M$ with cube side length $l$; $G_0 = \{g^i_0 \mid i=1,\dots,N_a\}$ is the set of initial 2D poses; $T_\text{max}$ is the maximum time horizon; $Q = \{q^i_j \mid i=1,\dots,N_a,\; j=1,\dots,N_q\}$ is the set of questions assigned to robots; and $Y = \{a^i_j \in \{\text{`A', `B', `C', `D'}\}\}$ is the ground-truth answer set (Zhang et al., 5 Feb 2026).

At time step $\xi:= (E, G_0, T_\text{max}, Q, Y),$0, robot $\xi:= (E, G_0, T_\text{max}, Q, Y),$1 has pose $\xi:= (E, G_0, T_\text{max}, Q, Y),$2, observes an RGB image $\xi:= (E, G_0, T_\text{max}, Q, Y),$3 and depth image $\xi:= (E, G_0, T_\text{max}, Q, Y),$4, and navigates using a collision-free planner $\xi:= (E, G_0, T_\text{max}, Q, Y),$5 that maps the current pose and a target position to the next feasible pose $\xi:= (E, G_0, T_\text{max}, Q, Y),$6. Robots may also send natural-language messages $\xi:= (E, G_0, T_\text{max}, Q, Y),$7 to one another. The benchmark objective is to maximize success rate while minimizing exploration time, with all answers required before $\xi:= (E, G_0, T_\text{max}, Q, Y),$8 (Zhang et al., 5 Feb 2026).

This formulation extends canonical EQA along several axes. It is multi-agent rather than single-agent, multi-task rather than one-question-per-episode, and explicitly fully cooperative: all robots know all questions, can share observations and answers, and optimize a common team objective. The paper further emphasizes that communication is not peripheral but required for avoiding redundant exploration and exploiting the advantages of multiple embodied agents (Zhang et al., 5 Feb 2026).

2. Benchmark construction and scenario design

The benchmark is built on the Habitat-Matterport 3D (HM3D) dataset and simulated in Habitat, yielding photo-realistic household environments with multiple rooms and varied object layouts (Zhang et al., 5 Feb 2026). The evaluation set contains 70 HM3D scenarios, while 20 additional HM3D scenarios are used to construct the calibration set for conformal prediction; those 20 scenes are not part of the evaluation metrics (Zhang et al., 5 Feb 2026).

Each evaluation scenario contains six task questions. The primary experimental configuration uses two robots, with three questions assigned to each robot, although all six questions are globally visible to both robots. The dataset therefore comprises 420 embodied question tasks across 70 scenarios. A three-robot variant is also used for scalability analysis (Zhang et al., 5 Feb 2026).

Scene complexity is stratified by floor area into three categories:

• Size 1: $\xi:= (E, G_0, T_\text{max}, Q, Y),$9
• Size 2: $E$0
• Size 3: $E$1

The benchmark is motivated by natural-language assignments such as “Turn off the TV if it is currently on” or “Bring the red pillow from the living room to the bedroom.” In the benchmark itself, these are operationalized as embodied question subproblems such as “Is the TV turned on?” or “What is the color of the pillow?” (Zhang et al., 5 Feb 2026). The paper conceptually motivates heterogeneous robots with non-transferable downstream tasks, but in the Habitat instantiation the embodied agents are homogeneous mobile robots with identical sensing and navigation models; heterogeneity is represented primarily through different assigned questions (Zhang et al., 5 Feb 2026).

3. Question taxonomy and annotation pipeline

The MM-EQA benchmark defines five types of embodied questions: location, identification, counting, existence, and state (Zhang et al., 5 Feb 2026). All questions are formatted as multiple choice with labels A, B, C, and D, even when the semantics are effectively binary.

Question type	Example
Location	“Where have I left the cushion? A) At the corner of the bedroom B) In the hallway C) Near the basketboard D) Next to TV in the living room”
Identification	“What bath mat is in the bathroom? A) Red B) White C) Black D) Gray”
Counting	“Did I leave any cues or balls on the pool table? A) None B) One C) Two D) Three”
Existence	“Have I put utensils and napkins on the dining table? A) Yes B) No”
State	“Is the washing machine turned on? A) Yes B) No”

Question generation is performed by GPT-4V, which is prompted with visual information from HM3D scenes to produce six embodied questions per scenario, corresponding answer options, and target objects relevant to each question (Zhang et al., 5 Feb 2026). Human annotators then refine the generated questions and answers to ensure correctness, discriminative options, and natural-language quality. The final ground-truth labels are stored as discrete choices $E$2 (Zhang et al., 5 Feb 2026).

The multi-task structure is part of the benchmark design rather than an incidental experimental detail. Each robot maintains question-specific confidence states and semantic maps, yet exploration is evaluated at the team level. This means that one robot may answer a question assigned to another robot if it becomes sufficiently confident, a property that distinguishes the benchmark from single-agent EQA task formulations (Zhang et al., 5 Feb 2026).

4. Observation, communication, and coordinated exploration

Each robot observes RGB-D data and a derived object list $E$3 produced by a VLM prompted to list the objects visible in the current image together with their colors (Zhang et al., 5 Feb 2026). Communication is natural-language, peer-to-peer, and decentralized: there is no central controller. Messages can include target-object requests, semantically relevant observations, approximate positions, and high-confidence answers for other robots’ questions (Zhang et al., 5 Feb 2026).

Exploration is organized around frontier-based mapping and semantic value maps. For a frontier point $E$4, the communication-based semantic value for robot $E$5 and question $E$6 is

$E$7

where $E$8 are weights and $E$9 counts relevant or target objects near $L \times W \times H$0. This is combined with a non-communication baseline as

$L \times W \times H$1

and then averaged across questions:

$L \times W \times H$2

These maps guide robots toward frontiers that are not only unexplored but semantically useful for the current question set (Zhang et al., 5 Feb 2026).

A question is answered when a VLM-based confidence check succeeds:

$L \times W \times H$3

where $L \times W \times H$4 is the answer probability for option $L \times W \times H$5 and $L \times W \times H$6 is a question-image relevance score (Zhang et al., 5 Feb 2026). A robot stops when it has answered all of its assigned questions, using its own observations and/or partner answers, or when the global time limit is reached.

The benchmark is tightly coupled to a communication setting in which message quality matters. CommCP uses conformal prediction to filter object-level relevance judgments before they are converted into messages, and the benchmark includes calibration scenes precisely for that purpose (Zhang et al., 5 Feb 2026). This suggests that MM-EQA is designed not merely as a navigation-and-QA dataset but as a testbed for decentralized communication policies under uncertainty.

5. Evaluation protocol and empirical behavior

An episode consists of one HM3D house, two or three robots, six total questions, globally known question text, robot-specific initial poses, and a shared exploration period beginning at time $L \times W \times H$7 (Zhang et al., 5 Feb 2026). During the episode, robots move, perceive, communicate, and periodically attempt to answer. The episode terminates when all assigned questions have been answered or when $L \times W \times H$8 is reached.

The primary metrics are Success Rate (SR) and Normalized Time Cost (NTC). SR is the proportion of correctly answered questions. NTC measures completion time, including both movement time and message-sending time, normalized to allow comparison across episodes; the paper reports SR-versus-NTC curves rather than a closed-form NTC equation (Zhang et al., 5 Feb 2026). Communication latency is explicitly varied through message-sending speeds of 0.25, 0.5, 1, 2, and 4 messages per second, with 1 message per second as the default (Zhang et al., 5 Feb 2026).

The main baselines are MMFBE (multi-agent multi-task frontier-based exploration without semantic guidance or communication), MMEuC (an extension of Explore Until Confident run independently per robot), and several ablations: Ours-No-CP, Ours-Com-Control, and Ours-No-Answer-Sharing (Zhang et al., 5 Feb 2026). In the reported results, CommCP reaches SR $L \times W \times H$9 at NTC $M$0, whereas MMFBE reaches SR $M$1 only at NTC $M$2. Average completion time is 445 seconds for CommCP versus 594 seconds for MMFBE (Zhang et al., 5 Feb 2026).

The benchmark is diagnostically sensitive to scene size and communication quality. The advantage of CommCP over MMFBE increases from modest in Size 1 scenes to an average NTC improvement $M$3 in Size 3 scenes (Zhang et al., 5 Feb 2026). Higher message rates accelerate early SR growth, but late-stage SR becomes similar once enough exploration has occurred. In the three-robot setting, calibrated communication improves efficiency, whereas removing conformal prediction can produce early performance drops because extra agents also amplify communication noise (Zhang et al., 5 Feb 2026). These analyses make the benchmark useful for studying not only end-task accuracy but also the interaction between latency, calibration, team size, and embodied exploration.

6. Position within the embodied QA landscape

Within embodied question answering more broadly, the MM-EQA benchmark occupies a distinct point in the design space. FAST-EQA targets single-agent embodied question answering on benchmarks such as HMEQA, EXPRESS-Bench, OpenEQA, and MT-HM3D, emphasizing question-conditioned region relevancy, bounded scene memory, and Chain-of-Thought reasoning over visual memory (Zhang et al., 17 Feb 2026). EXPRESS-Bench, by contrast, is an exploration-aware single-agent benchmark with 777 exploration trajectories and 2,044 question-trajectory pairs, and it introduces Exploration-Answer Consistency (EAC) to assess whether answers are grounded in the agent’s actual observations (Jiang et al., 14 Mar 2025). EQARewardBench evaluates reward models on OpenEQA-derived trajectories, focusing on scoring the quality of answer-and-reasoning traces rather than coordinating multiple embodied agents (Chen et al., 12 Jun 2025).

Relative to these benchmarks, MM-EQA is distinguished by four properties that are explicit in its definition: multi-agent embodiment, multi-task episodes, fully cooperative objectives, and natural-language communication as part of the benchmarked behavior (Zhang et al., 5 Feb 2026). It therefore shifts the central difficulty from single-agent search-and-answering to team-level division of labor, semantic message filtering, and answer sharing.

The benchmark also has clear limitations. It is confined to simulation in Habitat over HM3D scenes, not physical robots. It focuses on information gathering and question answering, not downstream manipulation. The question set is restricted to five categories, the communication channel is modeled through rate-based latency rather than explicit bandwidth or channel failures, and the evaluation corpus—70 test scenes and 420 questions—is substantial but still modest relative to large-scale vision-language datasets (Zhang et al., 5 Feb 2026). The paper identifies future directions in scaling to larger robot teams and environments, extending the semantics of the questions, and moving toward real-world deployment (Zhang et al., 5 Feb 2026).

In that form, the MM-EQA benchmark serves as a formalized benchmark for cooperative embodied information gathering in which success depends jointly on navigation, perception, communication, and confidence-aware answer production.