EmbRACE-3K: Embodied Reasoning Benchmark

• EmbRACE-3K is an embodied reasoning benchmark comprising over 3,000 language-guided tasks designed to evaluate exploration, dynamic spatial-semantic reasoning, and multi-stage goal execution.
• It employs a closed-loop data collection pipeline that integrates model-assisted instruction, human demonstrations, and step-wise chain-of-thought annotations to align perception, reasoning, and control.
• Empirical results reveal that while zero-shot models struggle with embodied tasks, targeted fine-tuning and reinforcement learning substantially improve performance and robustness.

EmbRACE-3K most explicitly denotes the 2025 embodied-AI dataset and benchmark introduced in “EmbRACE-3K: Embodied Reasoning and Action in Complex Environments,” which targets online interaction, active scene understanding, and language-conditioned decision making in photorealistic first-person environments (Lin et al., 14 Jul 2025). It consists of over 3,000 language-guided embodied tasks with about 26,000 decision steps, built in Unreal Engine using the UnrealCV-Zoo framework, and is organized to evaluate Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution (Lin et al., 14 Jul 2025). In the supplied literature, closely related nomenclature also appears in a mechanical-design context, where “EmBRACE-3K / PCA” refers to a prototype planetary cycloidal actuator based on 3K-H-V topology (Qi et al., 2022). The predominant research usage attached directly to the name, however, is the embodied reasoning benchmark.

1. Definition and research objective

EmbRACE-3K was introduced to address a mismatch between strong performance on passive, offline image and video understanding and limited effectiveness in embodied settings, where an agent must operate in a closed loop from a first-person perspective and where each action dynamically shapes subsequent observations (Lin et al., 14 Jul 2025). The benchmark is explicitly motivated by the observation that even state-of-the-art models such as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro struggle in open-environment interactions, with clear limitations in spatial reasoning and long-horizon planning (Lin et al., 14 Jul 2025).

The benchmark frames embodied intelligence as a joint problem of perception, reasoning, memory, and control. Its tasks span navigation, object manipulation, and multi-stage goal execution, and each task unfolds as a multi-step trajectory pairing first-person visual observations with high-level instructions, grounded actions, and natural language rationales that express the agent’s intent at every step (Lin et al., 14 Jul 2025). This design makes the benchmark simultaneously an evaluation suite and a supervised training resource.

A central feature is its emphasis on online interaction rather than static recognition. This distinction matters because the benchmark is constructed so that partial observability, egocentric viewpoint changes, and action-conditioned state transitions are not incidental noise but the core source of difficulty. This suggests that EmbRACE-3K is intended less as a generic multimodal dataset than as a diagnostic substrate for embodied failure analysis.

2. Dataset composition and task taxonomy

EmbRACE-3K contains over 3,000 language-guided embodied tasks with about 26,000 decision steps distributed across 24 diverse photorealistic Unreal Engine environments, including indoor and outdoor scenes (Lin et al., 14 Jul 2025). The environments are built using the UnrealCV-Zoo framework, and the dataset uses egocentric first-person views and diverse spatial layouts to facilitate real-world transferability (Lin et al., 14 Jul 2025).

Each step in a trajectory includes four principal annotations: the agent’s observation, a high-level instruction, the discrete action taken, and a “thinking” rationale, described as an explicit chain-of-thought explaining why the agent chose that action at that moment (Lin et al., 14 Jul 2025). The data format further includes an ordered sequence of egocentric images, the instruction in natural language, the action taken from a fixed set, the 6-DoF agent pose, and the step-wise rationale; all trajectories are normalized for consistency across length, with a maximum of 32 steps (Lin et al., 14 Jul 2025).

The task set is curated into several instruction categories:

• Basic: Target is visible and immediately reachable; minimal reasoning.
• Exploration: Target is initially out of view, requiring active search.
• Dynamic Spatial-Semantic: Requires understanding of spatial relations that change as the agent moves.
• Multi-Stage: Multiple sequential goals.
• Interaction: Direct manipulation, including opening doors and picking and dropping objects (Lin et al., 14 Jul 2025).

This taxonomy is important because it partitions embodied competence into separable regimes rather than treating success as a single scalar. The benchmark’s three headline dimensions—Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution—therefore function as high-level aggregations over task forms that stress different algorithmic bottlenecks (Lin et al., 14 Jul 2025).

3. Data construction pipeline and supervision signals

The data collection pipeline proceeds in four explicit stages. First, agent poses are sampled across scenes for diversity and reachability. Second, Gemini 2.5 Pro generates task instructions conditioned on local object layout and the desired reasoning skill. Third, humans execute the tasks to produce high-quality action sequences. Fourth, each action is paired with a natural language rationale that explains its intent and context (Lin et al., 14 Jul 2025).

This pipeline combines synthetic environment generation, model-assisted instruction authoring, human demonstrations, and dense step-wise reasoning annotation. The result is a closed-loop dataset in which perception, instruction following, action selection, and rationale production are aligned at the level of individual decisions rather than only at episode completion.

A notable property of the benchmark is the inclusion of explicit chain-of-thought supervision at every step. The paper reports that these step-wise reasoning annotations directly improve success rate and efficiency, and that explicit reasoning helps the model maintain context and improves alignment between perception and intent (Lin et al., 14 Jul 2025). A plausible implication is that EmbRACE-3K treats rationales not merely as interpretability artifacts but as operational supervision for intent persistence under egocentric drift.

The benchmark is also designed to expose characteristic embodied failure modes. The reported failure modes are short-sighted exploration, dynamic spatial-semantic drift, and target forgetting (Lin et al., 14 Jul 2025). Because these are attached to step-wise trajectories rather than only terminal scores, the dataset supports error analysis at the level of intermediate policy state.

4. Evaluation protocol and quantitative metrics

EmbRACE-3K evaluates models in both in-domain and out-of-domain settings. In-domain tasks are sampled from training-like environments, whereas out-of-domain tasks come from new, unseen environments and are used to assess generalization (Lin et al., 14 Jul 2025). The model input consists of the current first-person image, optionally plus up to five recent frames and the starting frame, together with the natural language instruction and action history (Lin et al., 14 Jul 2025).

The baseline suite includes GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL-7B in its original form, along with fine-tuned Qwen2.5-VL-7B variants: supervised fine-tuning only, supervised fine-tuning followed by reinforcement learning, and a “no-thinking” ablation that removes chain-of-thought inputs (Lin et al., 14 Jul 2025).

The benchmark uses several episode-level metrics:

• Success Rate (SR): percentage of tasks completed successfully under spatial constraints, for example within 300 cm of the target and issuing the Finish action.
• Goal Distance Error (GDE): Euclidean distance from the agent to the target at the end of the episode.
• Step-based Success weighted by Path Length (SSPL): path-efficiency metric.
• Steps: mean number of actions per episode.
• Timeout Rate (TR): percentage of episodes exceeding the step threshold without finishing (Lin et al., 14 Jul 2025).

The SSPL metric is defined as

$$\text{SSPL}_i = S_i \cdot \frac{s_i^{\text{opt}}}{\max(s_i, s_i^{\text{opt}})}$$

where $S_i$ is success, $s_i^{\text{opt}}$ is the shortest action path, and $s_i$ is the number of actions taken (Lin et al., 14 Jul 2025). This metric is significant because it penalizes policies that succeed only through inefficient wandering, which is especially relevant in exploration-heavy embodied tasks.

5. Empirical results, model behavior, and training effects

The headline empirical result is that zero-shot generalist VLMs perform poorly on embodied tasks in this benchmark. In zero-shot settings, all models achieve success rates below 20%, and average success rate for challenging tasks remains below 20%, often far lower (Lin et al., 14 Jul 2025). The paper identifies this as evidence of the current limitations of VLMs in interactive environments.

Selected out-of-domain results illustrate the gap between passive understanding and embodied competence:

Model	Task	OOD result
GPT-4o	Exploration	3.6% SR; 4017.8 cm GDE
Gemini 2.5 Pro	Exploration	9.1% SR; 2166.5 cm GDE
Qwen2.5-VL-origin	Exploration	0.0% SR; 9978.3 cm GDE
GPT-4o	Multi-stage	2.7% SR; 1312.9 cm GDE
Qwen2.5-VL-sft-rl	Exploration	30.9% SR; 1162.8 cm GDE
Qwen2.5-VL-sft-rl	Multi-stage	27.0% SR; 1265.7 cm GDE

These results are accompanied by several more specific observations. For Exploration in the out-of-domain setting, GPT-4o attains 3.6% success rate, Gemini 2.5 Pro 9.1%, and Qwen2.5-VL-original 0.0% (Lin et al., 14 Jul 2025). For Multi-stage tasks, GPT-4o reaches 2.7% success rate and Qwen2.5-VL-original again records 0.0% (Lin et al., 14 Jul 2025). Goal Distance Error for out-of-domain tasks exceeds 4000 cm for base models on Exploration and exceeds 8000–9000 cm for Multi-stage tasks (Lin et al., 14 Jul 2025).

Fine-tuning on EmbRACE-3K produces substantial gains. Supervised fine-tuning alone improves in-domain performance heavily, with success rate up to 81%, but generalizes poorly out of domain; for example, Exploration success rate drops from 71.4% in-domain to 22.8% out-of-domain (Lin et al., 14 Jul 2025). Supervised fine-tuning followed by reinforcement learning using the GRPO algorithm improves out-of-domain robustness, increasing Exploration success rate to 30.9% and Multi-stage success rate to 27.0% (Lin et al., 14 Jul 2025). The same training recipe also substantially reduces Goal Distance Error, including a reduction from approximately 9978.3 to 1162.8 in Exploration for Qwen2.5-VL-sft-rl (Lin et al., 14 Jul 2025).

The benchmark therefore functions both as a stress test and as a training resource. The reported improvements indicate that embodied reasoning deficits are not simply immutable limitations of contemporary VLM architectures; they are at least partially responsive to task-aligned supervision and reinforcement learning. At the same time, the sharp in-domain versus out-of-domain gap after SFT-only indicates that memorization of environment-specific regularities is insufficient for robust embodied generalization.

6. Naming, related usages, and potential confusion

The supplied literature contains several nearby strings that should be distinguished. The 2025 embodied benchmark appears under the title “EmbRACE-3K: Embodied Reasoning and Action in Complex Environments,” while the abstract and extracted summary also use the spelling “EmRACE-3K” (Lin et al., 14 Jul 2025). This variation is nominal rather than conceptual: both forms refer to the same benchmark of language-guided embodied tasks.

A separate extracted usage attaches “EmBRACE-3K / PCA” to a prototype planetary cycloidal actuator derived from a 3K-H-V robot-drive study (Qi et al., 2022). In that context, the design combines an involute 2K-H planetary input stage with a cycloidal K-H-V output stage, motivated by high stiffness, high accuracy, high loading, high efficiency, low backlash, compact size, and hollow structure (Qi et al., 2022). The reported prototype has dimensions of $\varnothing 82\,\mathrm{mm} \times 80\,\mathrm{mm}$, a $\varnothing 12\,\mathrm{mm}$ hollow shaft, a reduction ratio of 193.8:1, a rated output torque of 63 Nm, a torque density of 69 Nm/kg, and a measured forward efficiency of 65% (Qi et al., 2022). This is a mechanical-drive usage rather than an embodied-reasoning dataset.

The string “EMBRACE” also appears in radio astronomy as EMBRACE@Nançay, a dense aperture array prototype consisting of 4608 densely packed antenna elements creating a fully sampled, unblocked aperture for Square Kilometre Array-related work (Torchinsky et al., 2016). Likewise, “3k-4” appears in additive combinatorics in Freiman’s $3k-4$ theorem and in subsequent “$(3k-4)$-like” results for subset and subsequence sums (Balasubramanian et al., 2014, Mohan et al., 2024). These usages are unrelated to the embodied benchmark.

For research communication, the practical consequence is straightforward: when “EmBRACE-3K” is used without qualification, the most precise direct arXiv title match is the embodied reasoning benchmark (Lin et al., 14 Jul 2025), but adjacent literature demonstrates that the same or closely similar string may also appear in robotics hardware or in unrelated acronymic contexts.