WOLF-VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning

Abstract: Vision-Language-Action (VLA) models have recently demonstrated strong generalization in robotic manipulation, yet their applicability to whole-body, contact-rich humanoid locomotion remains severely underexplored due to data scarcity, the absence of dynamically consistent demonstrations, and the difficulty of encoding optimality and safety in learning-based pipelines. This work introduces a unified framework WOLF-VLA that integrates whole-body optimal-control (OC) motion synthesis with large-scale multi-modal dataset to train VLAs capable of generating humanoid locomotion policies directly from natural-language instructions. We construct a comprehensive dataset of dynamically feasible humanoid trajectories across six locomotion-related task families, each parameterized by environmental variations, object colors, placements, and visual distractors. We train a VLA model using the collected joint trajectories, ego-centric visual observations and natural language instruction, yielding a policy that exhibits strong reasoning and robustness to initial-condition variability, and competitive performance across several tasks and environment settings. A systematic ablation study demonstrates the impact of each modality on the model performance. The full dataset, model checkpoints, and benchmarking simulation suite will be openly released, establishing a reproducible dynamically consistent benchmark for whole-body humanoid locomotion rich VLA control and enabling future research in scalable transfer of instruction-driven locomotion policies.

• The paper introduces a scalable framework that synthesizes hundreds of hours of optimal, dynamically consistent humanoid locomotion trajectories using OC-based demonstrations.
• It employs Differential Dynamic Programming and transformer-based policy refinement to achieve high motion fidelity and robustness even in complex, contact-rich tasks.
• The framework outperforms state-of-the-art baselines by integrating multi-modal data, ensuring safe and adaptive long-horizon movement across diverse environments.

Whole-Body Humanoid Locomotion via Vision-Language-Action Framework: WOLF-VLA

Context and Motivation

The intersection of vision-language-action (VLA) models and whole-body humanoid robotics presents significant challenges—primarily due to the lack of large-scale, physically optimal datasets and the absence of integrated frameworks for contact-rich locomotion tasks. Previous efforts in VLA, while impactful for manipulation and fixed-base platforms, have not adequately addressed the complexities inherent in dynamic, multi-contact locomotion. WOLF-VLA aims to fill this gap by leveraging principled optimal control (OC) techniques for dataset generation and systematic policy learning, moving beyond teleoperated, non-optimal, or sparsely annotated data.

Framework Architecture and Dataset Construction

WOLF-VLA introduces a robust pipeline for generating and leveraging multi-modal demonstrations suited to high-dimensional humanoid locomotion:

• Dataset Generation: The framework employs OC solvers (specifically, Differential Dynamic Programming via Crocoddyl/Pinocchio) to synthesize hundreds of hours of optimal, dynamically consistent trajectories for the humanoid RH5 across six task families (forward walking, lateral walking, stair climbing, compound stair ascent/descent, rotation, squatting). Tasks are parameterized by spatial placements, object colors, and environmental distractors, creating systematic diversity for training and evaluation.
• Multi-Modal Data Structure: Each dataset sample encapsulates ego-centric visual observations (224x224 RGB), automatically generated NL instructions (with structured tags for spatial and task-specific cues), and corresponding OC joint trajectories. This alignment facilitates vision-language-action grounding in policy learning.
• Environment and Embodiment: Using gymnasium simulation, WOLF-VLA maintains embodiment-agnostic design principles, allowing re-targeting across platforms by constraining joint limits to match relevant hardware.

VLA Policy Learning Methodology

The learning paradigm is based on a transformer-architecture initialized from the GR00T-N1.5-3B policy. The visual backbone and language encoder are kept frozen, while action diffusion and projection layers are optimized. Sequence-modeling is framed as flow-matching: denoising sampled latent actions with respect to expert OC trajectories, leveraging a diffusion-based vector field inference. The policy operates over integrated proprioceptive, visual, and NL instruction tokens.

Experimental Evaluation and Benchmarking

WOLF-VLA is extensively evaluated across short, medium, and long-horizon tasks with rigorous ablation settings:

• Success Metrics: Binary success rates and soft success rates for compositional stair tasks, plus normalized joint range-of-motion (AROM) error rates (for hip, knee, and ankle) benchmark motion fidelity. Experiments involve up to 20 rollouts per task under both nominal and perturbed (with distractors) environments.
• Robustness: The learned policy exhibits resilience to initial condition variability and visual distractors; basic tasks maintain near-perfect success rates, while complex tasks (e.g., stair ascent/descent) demonstrate non-trivial robustness even as perceptual and coordination difficulty increases.
• Baseline Comparison: WOLF-VLA consistently outperforms state-of-the-art baselines (e.g., ACT, 70.5) trained on similar observation/action spaces. Baselines fail to achieve meaningful success rates, emphasizing the necessity of physically consistent demonstrations and integrated multi-modal encoding.
• Ablation Results: Removal of visual inputs results in catastrophic performance degradation, highlighting visual perception as the critical modality. Language and spatial instruction cues enhance generalization and compositional reasoning, but are secondary to vision. The paraphrasing experiment demonstrates robustness to NL instruction variations.

Numerical Results and Claims

• Motion Fidelity: AROM errors are consistently low across successful executions, confirming that trajectory patterns closely follow optimal OC demonstrations.
• Long-Horizon Control: The policy maintains stability and task completion over extended action sequences; WF tasks achieve success rates exceeding 95-100% across all horizons.
• Success Under Perturbation: Even with distractors, the trained policy retains substantial robustness, with only moderate declines for complex tasks.

Practical and Theoretical Implications

WOLF-VLA establishes a reproducible benchmark for instruction-driven whole-body locomotion using VLA models. The synthetic OC-based demonstration pipeline enables scalable generation of diverse, optimal, and safe motion profiles, facilitating high-quality policy transfer and standardized evaluation. Embodiment-agnostic dataset design enhances transferability across humanoid robot platforms. The results underscore the value of combining structured OC trajectories with multi-modal learning for physically plausible, robust locomotion.

Future Directions

Potential advancements include:

• Dataset Expansion: Synthesis of compounded locomotion and manipulation scenarios, further increasing environmental and task diversity.
• Long-Horizon Multi-Task Generalization: Investigation into architectures with extended temporal reasoning and multi-task compositionality.
• Real-World Deployment: Safe sim-to-real transfer on hardware platforms, leveraging the inherent optimality and safety of OC-generated data.
• Domain Adaptation: Use of photorealistic rendering and augmentation for improved robustness to sensory domain shifts.

Conclusion

WOLF-VLA marks a significant stride in learning vision-language-action policies for contact-rich, whole-body humanoid locomotion. It integrates optimal control-based motion generation with large-scale, multi-modal policy training, demonstrating superior motion fidelity, robustness, and successful transfer across task and environmental complexity. The framework establishes a scalable research benchmark and paves the way for advances in instruction-driven humanoid control, particularly relevant for safe, adaptive, real-world robotic deployment (2606.25591).