Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge (2512.06951v1)

Abstract: We present a vision-action policy that won 1st place in the 2025 BEHAVIOR Challenge - a large-scale benchmark featuring 50 diverse long-horizon household tasks in photo-realistic simulation, requiring bimanual manipulation, navigation, and context-aware decision making. Building on the Pi0.5 architecture, we introduce several innovations. Our primary contribution is correlated noise for flow matching, which improves training efficiency and enables correlation-aware inpainting for smooth action sequences. We also apply learnable mixed-layer attention and System 2 stage tracking for ambiguity resolution. Training employs multi-sample flow matching to reduce variance, while inference uses action compression and challenge-specific correction rules. Our approach achieves 26% q-score across all 50 tasks on both public and private leaderboards.

• The paper introduces a novel VLA model that integrates explicit task embeddings and stage tracking to resolve non-Markovian context in long-horizon tasks.
• It leverages mixed-layer attention and correlated Gaussian noise for flow matching to generate smooth, robust action trajectories.
• Experimental results on the BEHAVIOR Challenge show improved performance with a 26% mean partial success across 50 diverse household tasks.

Task Adaptation of Vision-Language-Action Models for Long-Horizon Robotics: An Expert Summary

Introduction and Problem Setting

This work presents a vision-language-action (VLA) model that achieved first place in the 2025 BEHAVIOR Challenge, a daunting benchmark focusing on long-horizon, diverse household tasks in highly realistic simulated environments. These tasks, often spanning thousands of timesteps and incorporating bimanual manipulation, navigation, and multimodal perception, pose fundamental difficulties: compounding errors, non-Markovian context, lack of recovery data, and multimodal action distributions. The approach adapts Pi0.5 (Intelligence et al., 22 Apr 2025)—a VLA model leveraging flow matching for continuous action prediction—with several architectural, training, and inference innovations to address these challenges.

Figure 1: The system processes images with SigLIP, extracts tokens using PaliGemma VLM, and predicts actions with a flow-matching action expert. Task embeddings and System 2 stage tracking resolve non-Markovian context.

Architectural Innovations

Task Embedding and Stage Tracking

Unlike Pi0.5’s language-based task specification, explicit trainable task embeddings are used for each of the 50 BEHAVIOR Challenge tasks, eliminating the need for text processing and sidestepping the generalization to novel tasks, which is not required in the challenge’s closed set. To resolve visual ambiguity and non-Markovian states, the model predicts fine-grained task stages via an auxiliary classifier (System 2), then fuses this stage signal with the task embedding, providing temporally coherent context and disambiguating visually similar yet semantically distinct task phases.

Figure 2: Stage tracking with a voting scheme provides robust non-Markovian context by smoothing over noisy, ambiguous predictions.

Mixed-Layer Attention via Learnable KV Transformation

A novel mixed-layer attention mechanism enables each action expert layer to attend to learned linear combinations of all VLM layers, parameterized by a small set of scalar weights and biases, rather than statically fixed one-to-one mappings. This adaptation increases architectural flexibility, allowing dynamic selection of visual-linguistic features most relevant for action generation.

Action Representation and Normalization

Actions are parameterized as timestamp-conditioned deltas (rather than absolute positions), normalized per timestamp, which controls for non-uniform action distributions across a predicted chunk. This design supports learning temporally smooth and invariant action trajectories, improving open-loop robustness.

Correlated Noise for Flow Matching and Correlation-Aware Inpainting

A primary contribution is the use of correlated Gaussian noise for flow matching. Rather than the isotropic $\mathcal{N}(0, \mathbf{I})$ noise used in standard flow matching, noise is sampled from $$\mathcal{N}(0, \beta\boldsymbol{\Sigma} + (1-\beta)\mathbf{I})$$ where $\boldsymbol{\Sigma}$ is the empirical block-structured action correlation matrix and $\beta=0.5$. This both regularizes learning and more closely matches the intrinsic structure of real robot actions, balancing the difficulty of denoising steps.

Figure 3: Empirical action correlation matrix exposes block-diagonal structure, supporting the choice of correlation modeling in action noise.

At inference, the temporal chunking of actions is supported with a rolling-window inpainting strategy—rather than treating initial “inpainted” timesteps independently, the approach conditions the free dimensions using the full empirical correlation, propagating constraints to produce continuous, physically plausible trajectories.

Figure 4: Soft, correlation-aware inpainting preserves smooth action sequences across chunk boundaries.

Training and Inference Procedures

Multi-Sample Flow Matching

To reduce gradient variance, the model predicts 15 flow-matching samples per VLM forward pass, amortizing the cost of expensive visual-linguistic feature extraction and stabilizing training.

Auxiliary FAST Training

The model employs the FAST tokenization strategy (Pertsch et al., 16 Jan 2025) with a small auxiliary loss weight, supporting learning of temporally compressed action distributions and regularization.

Action Compression

To accelerate execution, the 26 predicted actions per chunk are interpolated to 20 actual steps using cubic splines, effectively realizing a 1.3$\times$ speedup without increasing jitter or loss of control fidelity. Compression is adaptively disabled during gripper transitions to avoid degrading dexterous steps.

Figure 5: Cubic spline action compression enables significant execution speedup with maintained smoothness.

Correction Heuristics

To mitigate irrecoverable failures due to the lack of recovery data, simple correction rules are introduced—most notably, opening the grippers if closure is detected in states where demonstration data never closes the gripper. Such heuristics proved empirically to substantially elevate success rates on manipulation-heavy tasks.

Results and Analysis

Quantitative Results

The approach achieved a q-score (mean partial success) of $26\%$ averaged across all 50 tasks on both public and private leaderboards—a robust improvement over competing systems, which cluster below $20\%$ on the same metric.

Figure 6: Per-episode public evaluation scores: dark green denotes full success, light green partial, red denotes failure.

Failures are predominantly attributed to dexterity limitations in the underlying policy, compounded by the lack of recovery data, errors in program progress tracking, and confusion in novel, out-of-distribution states.

Figure 7: Failure cases are predominantly due to dexterity, followed by progress and state estimation errors.

Observations

The architecture demonstrates clear transfer: more diverse training (multi-tasking across all 50 tasks) induced emergent recovery strategies, albeit partial, which were entirely absent from single-task models. The effectiveness of correlated noise, action chunking with inpainting, and architectural flexibility are reflected in the efficiency and robustness of the best-performing policies.

Theoretical and Practical Implications

The correlation-aware methods described generalize to any scenario where the underlying action space has strong temporal and cross-dimensional structure, making these techniques relevant beyond household manipulation to manipulation-intensive mobile robotics and high-DOF agent control. The results confirm the importance of explicit context tracking (stage prediction) in long-horizon, visually ambiguous tasks. Moreover, the combination of learned policies with minimal, targeted heuristics appears a promising paradigm for robustification under data limitations.

Future Directions

Several promising avenues are suggested, including:

• Integration of recovery demonstrations to close the compounding error gap,
• Use of offline RL with advantage reweighting for robustification,
• Further leveraging vision-LLMs as “System 2” progress trackers,
• Incorporation of inherent safety and proprioceptive constraints for real-world transfer,
• Expansion of sensor coverage (e.g., fisheye cameras) for improved observation.

Conclusion

This work demonstrates that careful architectural adaptations—explicit action correlation modeling, mixed-layer multimodal attention, stage-aware context fusion—combined with pragmatic optimizations and minimal, targeted heuristics, can materially improve long-horizon task performance in complex, photo-realistic settings. The practical dominance of pretraining and transfer, multi-task-induced robustness, and hybrid learning-heuristic architectures are underscored. These results reinforce the viability of unified, scalable embodiment models for the next generation of advanced robotic control systems.

---

Reference: "Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge" (2512.06951)