2025 BEHAVIOR Challenge Overview

• The 2025 BEHAVIOR Challenge is a benchmark that evaluates long-horizon embodied AI by testing agents on 50 diverse, realistic household tasks using the BEHAVIOR-1K framework and OmniGibson simulation.
• Participants leverage 10,000 expert demonstrations and state-of-the-art vision-language-action models to address coordinated navigation, manipulation, and context-aware decision making.
• Empirical results highlight significant partial progress via q-scores while revealing unresolved challenges in dexterity, perception fidelity, and recovery from ambiguous task stages.

Searching arXiv for the 2025 BEHAVIOR Challenge and closely related benchmark papers to ground the article in the current literature. The 2025 BEHAVIOR Challenge was a competition benchmark for long-horizon embodied agents in realistic household simulation. It was built on BEHAVIOR-1K, a human-centered benchmark of everyday activities, and operationalized evaluation on a fixed set of 50 household tasks in photo-realistic simulation. The challenge asked a single policy to handle navigation, bimanual manipulation, context-aware decision making, and long temporal horizons, with ranking based primarily on q-score, which measures partial goal completion across tasks, while binary full-task success was also reported (Li et al., 2024, Larchenko et al., 7 Dec 2025).

1. Origins and scope

The challenge inherits its problem setting from BEHAVIOR-1K, which was designed around what people actually want robots to do in everyday life rather than around a researcher-selected toy task suite. BEHAVIOR-1K contains 1,000 everyday activities grounded in 50 scenes and more than 9,000 objects with rich physical and semantic properties, and its activity inventory was derived from a survey of 1,461 participants ranking around 2,090 candidate activities (Li et al., 2024).

Within that broader substrate, the 2025 challenge instantiated 50 representative tasks. The task family is explicitly long-horizon. One competition report describes average episode lengths of about 6.6 minutes, with some tasks averaging 14 minutes, and emphasizes that the setting requires coordinated use of a mobile base, two 7-DOF arms, and parallel-jaw grippers under multi-camera RGB observations from head and wrist cameras (Larchenko et al., 7 Dec 2025). Another report notes that many tasks span several hundred frames and often require 5–10 distinct skills, sometimes more than 12, especially when navigation and manipulation must be interleaved repeatedly across rooms and containers (Bai et al., 10 Dec 2025).

This combination of duration, embodiment, and task diversity is central to the challenge’s identity. It is not merely a manipulation benchmark, nor merely a navigation benchmark. It is a stress test of skill chaining, temporal credit assignment, recovery, and goal-conditioned execution in realistic domestic environments.

2. Benchmark substrate: BEHAVIOR-1K and OmniGibson

BEHAVIOR-1K supplies both the task semantics and the simulated world model. Its 50 scenes span houses, houses with gardens, hotels, offices, grocery stores, halls, restaurants, and schools, and its object inventory covers more than 1,900 categories with annotations such as cookable, sliceable, fillable, openable, toggleable, deformable, liquid, cloth, rope, flammable, heatSource, particleSource, foldable, and broken (Li et al., 2024).

Those annotations are not decorative metadata. They determine which activities are physically meaningful and logically satisfiable. BEHAVIOR-1K tasks are defined in a BDDL-like logical format with objects, initial conditions, goal conditions, predicates, transition rules, and scene-room placement constraints. The benchmark typically requires 3–47 objects per activity and 2–11 state changes per activity. To support these activities, the underlying simulator, OmniGibson, models rigid bodies, deformable bodies, liquids, cloth-like materials, thermal effects, and complex state transitions; the benchmark paper states that without these capabilities, over half of BEHAVIOR-1K activities would not be simulatable (Li et al., 2024).

The 2025 challenge therefore rested on an unusually broad notion of embodied competence. Success depended not only on reaching and grasping, but also on operating articulated objects, handling multi-object workflows, satisfying symbolic goal predicates, and coping with physically realistic side effects of interaction.

3. Task formulation, demonstrations, and evaluation

The challenge provided 10,000 teleoperated expert demonstrations for the 50 challenge tasks, corresponding to 200 demonstrations per task and more than 1,200 hours, together with multimodal observations and skill annotations (Bai et al., 10 Dec 2025). The goal was not subtask recognition or short-horizon imitation in isolation. Policies were evaluated in simulation by whether they satisfied full task goals defined through BDDL predicates, so an agent had to complete the household activity rather than merely execute a plausible fragment (Bai et al., 10 Dec 2025).

The official ranking metric was q-score. A first-place report states that q-score averages partial goal completion across episodes and tasks rather than requiring only binary success, and that each task is evaluated on 10 episodes, with public scores computed on 10 published instances and private scores on another 10 held-out instances per task (Larchenko et al., 7 Dec 2025). Binary success was also reported, but it was not the primary ranking metric.

This distinction is important because the benchmark was deliberately constructed so that partial progress is common even when full completion remains rare. A policy might navigate correctly, manipulate several objects correctly, or satisfy part of a compound goal while still failing the task as a whole. Q-score captures that graded progress; full-task success does not.

4. Leaderboard outcomes and what they measured

The standard-track leaderboard reported by the second-place system is summarized below (Bai et al., 10 Dec 2025).

Rank	Team	Test Q-score
1	Robot Learning Collective	0.2599
2	Comet	0.2514
3	SimpleAI Robot	0.1591
4	The North Star	0.1204

A separate first-place report gives essentially the same top result as about 0.2605 public q-score and 0.2599 private q-score, with binary success around 11–12% (Larchenko et al., 7 Dec 2025). The second-place report gives Comet a test q-score of 0.2514 and describes it as very close to the winner while substantially ahead of the rest of the field (Bai et al., 10 Dec 2025).

A common misconception is to read a q-score near 0.26 as meaning that roughly a quarter of tasks were fully solved. The reports show that this is incorrect. The winning system’s binary success remained only around 11–12%, while q-score was higher because it rewards partial goal completion (Larchenko et al., 7 Dec 2025). The competition therefore exposed both progress and unsolved difficulty: top systems could often make substantial headway on long activities, but full completion remained infrequent.

5. Dominant policy architectures and solution strategies

The two most detailed competition solutions both started from $\pi_{0.5}$, a vision-language-action model with a flow-matching action head, but adapted it differently to the challenge (Larchenko et al., 7 Dec 2025, Bai et al., 10 Dec 2025).

The first-place system replaced open-ended language conditioning with 50 trainable task embeddings, one per task, because the benchmark has a fixed task set. Its perception stack used a frozen SigLIP-So400m/14 encoder on three RGB views, followed by a PaliGemma-based backbone and an 18-layer Gemma-style action expert that predicts a length-30 chunk of 23D continuous actions by flow matching. Its principal algorithmic contribution was correlated noise for flow matching: instead of sampling $$\boldsymbol{\epsilon} \sim \mathcal{N}(0,\mathbf{I})$$, it estimated an empirical action covariance and shrank it toward identity,

$$\boldsymbol{\Sigma}_{\text{reg}} = \beta \boldsymbol{\Sigma} + (1-\beta)\mathbf{I},$$

with $\beta=0.5$, so the noisiest training inputs still preserved realistic action correlations. The same correlation structure was then used for correlation-aware inpainting at chunk boundaries, alongside learnable mixed-layer attention, a stage classifier for “System 2” stage tracking, and multi-sample flow matching with $N=15$ samples per observation (Larchenko et al., 7 Dec 2025).

The second-place system emphasized scaling and post-training. It compared pretraining on 1, 7, 10, and 50 tasks, denoted pt1, pt7, pt10, and pt50, and reported that broader pretraining coverage improved generalization and increased the number of tasks on which successful rollouts were obtained. After pretraining it reached a validation q-score of 0.19. It then applied iterative Rejection Sampling Fine-Tuning rather than online RL: start from human demonstrations, perturb initial robot poses, roll out the current policy, keep only successful rollouts, add them to the dataset, retrain, and repeat for $N=3$ rounds. The report states that this raised validation q-score to 0.22, while a union-over-checkpoints upper bound reached 0.31, indicating substantial remaining headroom (Bai et al., 10 Dec 2025).

Despite their differences, both solutions treated the challenge as a long-horizon adaptation problem for pretrained foundation policies rather than as a from-scratch RL problem. Both reports argue, in different ways, that broad prior data coverage and carefully engineered post-training are more effective than naive online optimization in this regime.

6. Empirical lessons, bottlenecks, and open problems

The strongest empirical findings concern control stability, perception fidelity, and recovery. In the second-place report, open-loop control variants failed on the “turning-on-radio” task, with Temporal Ensemble and Receding Temporal both scoring 0.00, while closed-loop Receding Horizon reached 0.25. Action horizon showed a non-monotonic effect: 8 scored 0.00, 16 scored 0.10, 50 scored 0.25, and 32 scored 0.30. High-resolution visual input mattered strongly, with 224×224 head/wrist images scoring 0.30 and 720×480 scoring 0.60 in the reported ablation. Absolute Joint action representation outperformed Delta Joint, 30 Hz outperformed 15 Hz, and removing proprioceptive state input hurt performance (Bai et al., 10 Dec 2025).

The first-place report emphasizes a different but complementary bottleneck: partial observability and ambiguity across hidden task stages. Visually similar observations can correspond to different logical stages, such as holding an object at the start of a task versus returning it later. Its stage tracker divided each task into roughly 5–15 stages and used majority-vote rules over the last three predictions to advance, skip, or roll back stages. The same report also describes challenge-specific correction rules, particularly a gripper recovery heuristic that treated certain unseen closed-gripper states as failed grasps and reopened the gripper; the authors state that this approximately doubled success on a subset of tasks and had a very large effect on final score (Larchenko et al., 7 Dec 2025).

Both papers are explicit that the challenge remains far from solved. The winning report states that the demonstrations contain only successful trajectories, so the policy rarely sees recovery states, and estimates that the model effectively saw only about two epochs over the data during training. It identifies dexterity and clumsiness, progress and order mistakes, ambiguous states, and gripper failures as dominant failure modes (Larchenko et al., 7 Dec 2025). The second-place report likewise argues that long-horizon robustness remains unresolved and recommends DAgger, on-policy distillation, and RL methods that combine positive and negative reward signals to improve sample efficiency (Bai et al., 10 Dec 2025).

Taken together, these findings define the 2025 BEHAVIOR Challenge as a benchmark where partial progress by pretrained VLAs is measurable, but robust household autonomy is not yet routine. The challenge demonstrated that broad pretraining, offline-friendly post-training, closed-loop execution, and high-fidelity perception can move the frontier. It also showed that recovery, dexterity, and state coverage remain the limiting factors for long-horizon embodied AI in realistic domestic simulation (Li et al., 2024).