Multi-Agent Multi-Task Embodied QA

• The paper introduces MM-EQA, a framework that integrates decentralized planning, perception, and a calibrated LLM communication mechanism for efficient embodied question answering.
• It details a conformal prediction-based calibration routine that ensures only reliable messages guide the agents’ exploration in a shared 3D environment.
• Empirical evaluations on photorealistic HM3D scenes demonstrate that calibrated communication reduces exploration time while significantly improving answer accuracy compared to non-cooperative approaches.

Multi-Agent Multi-Task Embodied Question Answering (MM-EQA) is a fully cooperative embodied QA setting in which multiple agents explore a shared 3D environment, answer distinct embodied questions under a finite horizon, and coordinate through decentralized natural-language communication so as to maximize answer correctness while minimizing exploration time. In the formulation introduced by "CommCP: Efficient Multi-Agent Coordination via LLM-Based Communication with Conformal Prediction," MM-EQA is a novel extension of canonical Embodied Question Answering (EQA), motivated by deployments with multiple heterogeneous robots that must interpret human assignments, gather information, and avoid redundant search (Zhang et al., 5 Feb 2026). A related line of work on Multi-Embodied LLM Explorers (MELE) studies multi-agent embodied QA through independent exploration and downstream answer aggregation, providing a complementary perspective on how multiple LLM-based agents can answer questions grounded in exploration trajectories (Patel et al., 2024).

1. Formalization of the MM-EQA problem

The MM-EQA formulation specifies a set of agents $A=\{1,2,\dots,N_a\}$, a single 3D environment $E$ of real-world dimensions $L\times W\times H$ discretized into a voxel map $M$, and a time horizon $T_{\max}$. Each agent $i\in A$ starts at an initial pose $g_0^i\in SE(2)$. The question set is

$Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$

where $q_j^i$ is the $j^{\text{th}}$ multiple-choice question assigned to agent $E$0. Each question has four possible answers in $E$1, and the ground-truth answers are collected in

$E$2

At time $E$3, each agent $E$4 receives an RGB image $E$5 and a depth image $E$6. The collective pose vector is denoted $E$7. Each agent has a low-level controller and a collision-free planner $E$8 such that, given current pose $E$9 and a 2D waypoint $L\times W\times H$0, the planner yields the next pose

$L\times W\times H$1

Communication is explicit: at any time $L\times W\times H$2, agent $L\times W\times H$3 may send a natural-language message $L\times W\times H$4 to its peers (Zhang et al., 5 Feb 2026).

The objective is to learn decentralized policies for perception, planning, and communication,

$L\times W\times H$5

for each $L\times W\times H$6, such that all agents jointly navigate the scene, answer each $L\times W\times H$7 by time $L\times W\times H$8, and maximize the total number of correct answers while minimizing total exploration time. The source text summarizes this as a decentralized cooperative POMDP with natural-language messaging. This suggests that MM-EQA is not merely a multi-robot navigation problem or a standard QA problem, but a coupled optimization over information acquisition, semantic interpretation, and inter-agent coordination (Zhang et al., 5 Feb 2026).

2. CommCP: calibrated decentralized communication

CommCP augments each agent policy $L\times W\times H$9 with three submodules: LLM-based message generation, a conformal-prediction-based calibration layer, and a decentralized speaking protocol. The communication module is triggered when an agent observes detected objects

$M$0

where $M$1 is an object label with color $M$2, and receives a request $M$3 from agent $M$4 for assistance on question $M$5 with target object(s) $M$6. The agent then prompts an LLM with a zero-shot chain-of-thought template. The system prompt is: “As a robot in a house, your partner is looking for $M$7. You can inform them about what you have observed.” The user prompt is: “You observe $M$8. Your partner wants to find $M$9. … Provide your analysis.” From this analysis, a second prompt elicits a single-token answer in $T_{\max}$0, and the LLM returns a probability $T_{\max}$1 for each $T_{\max}$2, thereby defining a categorical distribution over the four options for each observed object (Zhang et al., 5 Feb 2026).

To ensure that only reliable judgments are communicated, CommCP uses split conformal prediction on the option probabilities. Let $T_{\max}$3 be a calibration set for label $T_{\max}$4, with $T_{\max}$5 ignored because they are discarded. Under the exchangeability assumption, CommCP computes the $T_{\max}$6 quantile

$T_{\max}$7

At test time, for a new pair $T_{\max}$8, the conformal prediction set is

$T_{\max}$9

By standard conformal guarantees,

$i\in A$0

Only if $i\in A$1 is the corresponding object reported as “relevant” (Option B) or “target” (Option A). The final message $i\in A$2 is instantiated as: “I see $i\in A$3 that may be relevant to your target $i\in A$4, and $i\in A$5 may be at position $i\in A$6.” The stated purpose is to minimize receiver distractions and enhance communication reliability (Zhang et al., 5 Feb 2026).

The decentralized speaking protocol applies this pipeline at each time step. For each unanswered request $i\in A$7, an agent observes $i\in A$8, prompts the LLM, applies the conformal threshold $i\in A$9, marks objects as target or relevant when $g_0^i\in SE(2)$0, assembles $g_0^i\in SE(2)$1 if any target or relevant object is found, and broadcasts it. The agent then uses received messages $g_0^i\in SE(2)$2 to update semantic values and plans the next waypoint by

$g_0^i\in SE(2)$3

Agents also share full answers: if agent $g_0^i\in SE(2)$4’s VLM yields answer probabilities $g_0^i\in SE(2)$5 and a relevance score $g_0^i\in SE(2)$6 on $g_0^i\in SE(2)$7, then if there exists a unique $g_0^i\in SE(2)$8 with $g_0^i\in SE(2)$9, agent $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$0 broadcasts “My answer to $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$1 is $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$2.” This mechanism directly targets redundancy reduction during cooperative exploration (Zhang et al., 5 Feb 2026).

3. Benchmark construction and evaluation protocol

The MM-EQA benchmark introduced alongside CommCP uses photorealistic indoor scenes from HM3D. The benchmark is described as containing 70 photorealistic indoor scenes from HM3D, split into 50 train, 20 calibration (for conformal), 70 test. In each scene, six multiple-choice embodied questions are generated via GPT-4V plus human refinement. The task types are location, identification, counting, existence, and state, with examples including “Where is the red cushion?”, color/material identification, number of objects, yes/no existence, and on/off state queries. For multi-agent assignment, two robots are used per test scene, each assigned three distinct questions; the robots share observations but cannot swap assignments (Zhang et al., 5 Feb 2026).

Evaluation uses two primary metrics. Success Rate (SR) is

$Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$3

Normalized Time Cost (NTC) is defined using the time $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$4 when all questions are answered or the horizon is reached: $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$5 Communication overhead is measured implicitly by varying the message-sending rate $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$6 msg/s and observing its impact on SR versus NTC. These choices emphasize joint accuracy-efficiency tradeoffs rather than answer quality alone (Zhang et al., 5 Feb 2026).

A notable aspect of the benchmark is that it explicitly binds question answering to embodied information gathering across multiple concurrent tasks. This suggests a departure from settings where agents answer from static observations or shared global maps. The benchmark instead requires agents to allocate exploration effort across distinct assignments while still exploiting shared semantic information (Zhang et al., 5 Feb 2026).

4. Empirical behavior, ablations, and scaling properties

In two-robot teams, CommCP achieves $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$7 at $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$8, compared with $Q = \{\,q_j^i \mid i\in A, j=1\dots N_q\,\},$9 for the best non-communicative baseline, MMFBE. The reported overall completion time is 445 s for CommCP versus 594 s for MMFBE. The no-communication baseline MMEuC underperforms, supporting the claim that independent “explore until confident” behavior is less efficient than calibrated coordination (Zhang et al., 5 Feb 2026).

The ablation studies isolate the role of message quality. Ours-No-CP, which uses uncalibrated messages, collapses to MMEuC performance due to misleading information. Ours-Com-Control, which uses random fixed-size messages, fails to match calibrated-CP quality. Ours-No-Answer-Sharing suffers higher NTC, since agents redundantly explore their partners’ questions. These results directly counter a common misconception that any additional communication is beneficial: the reported findings instead indicate that the utility of communication depends on statistical reliability and task-relevant content (Zhang et al., 5 Feb 2026).

The communication-latency study varies $q_j^i$0 and reports that higher message rates speed up early SR growth, while beyond a threshold all curves converge. Across all tested $q_j^i$1, CommCP outperforms MMFBE. The scalability study extends to a three-robot team, where CommCP again yields the fastest SR-versus-NTC curve and retains the benefits of conformal prediction, whereas Ours-No-CP degrades as spurious messages increase. In a scene-size study, large scenes are defined as having area $q_j^i$2, and CommCP reduces NTC by $q_j^i$3 relative to MMFBE. The paper interprets this as robustness in complex environments (Zhang et al., 5 Feb 2026).

Taken together, these results characterize MM-EQA as a regime in which coordination quality dominates raw communication volume. A plausible implication is that message calibration plays the role of a bandwidth filter as much as a confidence estimator, since the principal failure mode of uncalibrated communication is not silence but distraction (Zhang et al., 5 Feb 2026).

5. Relation to MELE and aggregation-based embodied QA

A related formulation appears in "Multi-LLM QA with Embodied Exploration," which models embodied QA in an unknown environment using Multi-Embodied LLM Explorers (MELE). The environment is formalized as a partially observable MDP

$q_j^i$4

and $q_j^i$5 independent LLM-based agents $q_j^i$6 receive queries $q_j^i$7 and must output an answer $q_j^i$8, with $q_j^i$9 in the reported experiments. Each agent explores over a horizon $j^{\text{th}}$0, collects a trajectory

$j^{\text{th}}$1

and constructs an evidence set $j^{\text{th}}$2. In answer mode, the LLM is prompted with summarized observations and the question, and returns a distribution $j^{\text{th}}$3 (Patel et al., 2024).

MELE studies three aggregation methods for producing a final answer $j^{\text{th}}$4: debating, majority voting, and a Central Answer Module (CAM). Debate uses a fixed-length, turn-based exchange and then a majority vote over final agent votes. Majority voting directly computes the label with the most votes. CAM trains a parametric classifier $j^{\text{th}}$5 by cross-entropy on labeled data and predicts by $j^{\text{th}}$6. On a 215-node Matterport3D environment with a 95%/5% split and mean$j^{\text{th}}$7std over 5 seeds, the reported accuracies are 60.2\%$j^{\text{th}}$81.1\% for majority vote, 65.3\%$j^{\text{th}}$91.4\% for debate, and 88.9\%$E$000.8\% for CAM (XGBoost). The paper describes this as a 46% higher accuracy for CAM compared against the other non-learning-based aggregation methods (Patel et al., 2024).

Within that framework, multi-tasking is supported by enriching the question encoding with a task token such as COUNT, FIND, or STATE and adding a task-specific prefix in the system prompt. The task categories explicitly include object presence, object counting, location queries, and state inference. This is adjacent to MM-EQA rather than identical to CommCP’s formulation: MELE emphasizes independent exploration and answer aggregation, whereas CommCP emphasizes decentralized natural-language communication among active embodied agents. This suggests two complementary design axes for MM-EQA systems: calibrated inter-agent messaging during exploration and learned aggregation after exploration (Patel et al., 2024).

6. Limitations, misconceptions, and open research problems

The limitations reported for CommCP concentrate on calibration, distribution shift, model choice, protocol richness, and scaling. Only 20 HM3D scenes were used for calibration data; the paper notes that larger and more diverse calibration sets would tighten coverage guarantees. The conformal procedure relies on an I.I.D. exchangeability assumption, while real-world scene distributions may violate exchangeability, for example through correlated object layouts; more sophisticated CP variants such as Mondrian conformal are therefore identified as relevant. Because probability outputs are required, the study uses LLaMA3-8B rather than closed-source giants, and it explicitly suggests that better open-source or hybrid LLM+VLM pipelines would likely improve message quality. The prompts are one-shot rather than interactive, leaving multi-hop reasoning and dialog as an extension. Scaling from 2–3 robots to larger, potentially heterogeneous teams with conflicting objectives is left open, as is incorporating explicit bandwidth and delay penalties into decentralized protocol optimization (Zhang et al., 5 Feb 2026).

The MELE line exposes a different set of limitations. Its current queries are binary yes/no and assume static environments; supervised training of CAM requires labeled questions in each environment; and answer quality depends on the scene-understanding frontend, with GLIP given as an example. Future directions include larger teams with dynamic agent reliability tracking, open-ended and subjective questions, real-world deployment with noisy sensors and non-stationary objects, and transfer of CAM to video-QA or instruction following (Patel et al., 2024).

Several misconceptions can be rejected directly from the reported evidence. First, more communication is not automatically better: CommCP’s ablations show that uncalibrated or random communication does not match calibrated CP-based communication and can collapse to no-communication performance (Zhang et al., 5 Feb 2026). Second, independent multi-agent exploration is not sufficient for strong embodied QA: both CommCP and MELE report gains from explicit coordination mechanisms, whether through calibrated messaging or learned aggregation (Zhang et al., 5 Feb 2026). Third, MM-EQA should not be reduced to a pure QA benchmark. The formalism, metrics, and experimental design all treat navigation, observation, answer confidence, and time cost as coupled variables. A plausible implication is that future progress in MM-EQA will depend less on isolated LLM answer quality than on joint optimization of embodied exploration, uncertainty calibration, and distributed decision-making (Zhang et al., 5 Feb 2026).