A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation

Abstract: Robot manipulation has seen tremendous progress in recent years, with imitation learning policies enabling successful performance of dexterous and hard-to-model tasks. Concurrently, scaling data and model size has led to the development of capable language and vision foundation models, motivating large-scale efforts to create general-purpose robot foundation models. While these models have garnered significant enthusiasm and investment, meaningful evaluation of real-world performance remains a challenge, limiting both the pace of development and inhibiting a nuanced understanding of current capabilities. In this paper, we rigorously evaluate multitask robot manipulation policies, referred to as Large Behavior Models (LBMs), by extending the Diffusion Policy paradigm across a corpus of simulated and real-world robot data. We propose and validate an evaluation pipeline to rigorously analyze the capabilities of these models with statistical confidence. We compare against single-task baselines through blind, randomized trials in a controlled setting, using both simulation and real-world experiments. We find that multi-task pretraining makes the policies more successful and robust, and enables teaching complex new tasks more quickly, using a fraction of the data when compared to single-task baselines. Moreover, performance predictably increases as pretraining scale and diversity grows. Project page: https://toyotaresearchinstitute.github.io/lbm1/

• The paper demonstrates that finetuned LBMs achieve comparable or superior performance with 3–5x less task-specific data than single-task models.
• The paper reveals that LBMs generalize better and are more robust to distribution shifts, as shown by significant improvements in both simulated and real-world tasks.
• The paper highlights that performance improves smoothly with increased diverse pretraining data, underscoring the benefits of large-scale multitask pretraining.

A Critical Evaluation of Large Behavior Models for Multitask Dexterous Manipulation

This paper presents a comprehensive empirical study of Large Behavior Models (LBMs) for multitask dexterous robot manipulation, focusing on the effects of large-scale multitask pretraining and rigorous evaluation protocols. The authors extend the Diffusion Policy paradigm to train and analyze LBMs on a diverse corpus of both simulated and real-world manipulation data, with a particular emphasis on statistical rigor and reproducibility in evaluation.

Methodological Overview

The core contribution is a systematic comparison between multitask-pretrained LBMs and single-task baselines across a wide range of manipulation tasks, both in simulation and on physical hardware. The LBMs are trained on approximately 1,700 hours of demonstration data spanning over 500 tasks, sourced from internal teleoperation, simulation, and public datasets. The policy architecture is a Diffusion Transformer conditioned on vision, language, and proprioception, outputting continuous action sequences.

The evaluation protocol is notable for its scale and rigor: over 1,800 real-world rollouts and 47,000 simulation rollouts are conducted using blind, randomized A/B testing, with careful control of initial conditions and robust statistical analysis. Performance is measured using both binary success rates and a task completion metric based on intermediate milestones, with Bayesian posteriors and hypothesis testing used to assess statistical significance.

Key Empirical Findings

The study yields several strong and quantifiable results:

• Data Efficiency: Finetuned LBMs require 3–5x less task-specific data to match the performance of single-task models. In some real-world tasks, LBMs finetuned with only 15% of the available data statistically outperform single-task models trained on the full dataset.
• Generalization and Robustness: Finetuned LBMs consistently outperform single-task baselines on both "seen" (pretraining) and "unseen" (novel) tasks, with the performance gap widening under distribution shift (e.g., changes in lighting, object appearance, or robot station). For example, in simulation under distribution shift, finetuned LBMs are statistically superior in 10/16 "seen" tasks and all "unseen" tasks.
• Scaling Laws: Performance of finetuned LBMs improves smoothly with increasing pretraining data, with no evidence of sharp inflection points or saturation at the examined data scales. There is a clear tradeoff: more diverse pretraining data is especially beneficial when task-specific finetuning data is limited.
• Non-finetuned LBM Performance: Pretrained LBMs without task-specific finetuning achieve nonzero success rates on "seen" tasks but do not consistently outperform single-task models. The authors attribute this to limitations in language conditioning and model capacity.

Experimental Design and Implementation Details

The policy architecture is a Diffusion Transformer (DiT) conditioned on CLIP-based vision and language features, as well as proprioceptive state. The model predicts 16-step, 20-dimensional action sequences at 10 Hz, with deployment executing the first 8 steps before replanning. Pretraining uses a batch size of 2560 and a learning rate of 3e-4, while finetuning uses a batch size of 320 and a learning rate of 2e-5. Data normalization and filtering are carefully analyzed, with the authors noting that such preprocessing decisions can have effects comparable to architectural changes.

The evaluation protocol is designed to minimize bias and maximize reproducibility. In simulation, initial conditions are controlled via random seeds; in hardware, operators use image overlays to match scenes. All evaluations are blind and randomized, with statistical significance assessed via Bayesian posteriors and pairwise hypothesis testing (with Bonferroni correction).

Implications and Limitations

The results provide strong empirical support for the LBM/foundation model paradigm in robotics: large-scale multitask pretraining followed by task-specific finetuning yields policies that are more data-efficient, robust to distribution shift, and capable of generalizing to novel tasks. The findings also highlight the importance of rigorous, high-throughput evaluation protocols and careful statistical analysis, as many effects are only detectable with large sample sizes and controlled conditions.

However, the study also identifies several limitations:

• Language Conditioning: The current LBMs use relatively small language encoders (CLIP), and language steerability remains a bottleneck for zero-shot generalization and multitask performance.
• Evaluation Noise: Despite careful protocols, real-world evaluation is subject to human error and environmental variability, potentially masking small effects.
• Training Stochasticity: The analysis does not account for variance across multiple training runs with different random seeds, which could affect the robustness of conclusions about model superiority.

Future Directions

The work suggests several avenues for further research:

• Scaling Language and Multimodal Capacity: Integrating larger, more capable language and vision encoders (e.g., VLA models) may improve language steerability and zero-shot performance.
• Automated and Standardized Evaluation: Developing more automated, scalable, and standardized real-world evaluation frameworks will be critical for benchmarking future generalist robot policies.
• Data and Preprocessing Analysis: Systematic studies of data normalization, filtering, and augmentation are needed, as these factors can have outsized effects on policy performance.
• Sim2Real Transfer: The co-training of simulation and real-world data, and the use of domain randomization and system identification, remain important for improving sim2real transfer and robustness.

Conclusion

This paper provides a rigorous empirical foundation for the use of Large Behavior Models in multitask dexterous manipulation. The demonstrated data efficiency, robustness to distribution shift, and generalization capabilities of finetuned LBMs underscore the practical value of the foundation model approach in robotics. The methodological emphasis on statistical rigor and reproducibility sets a high standard for future empirical work in the field. The results also caution that preprocessing and evaluation design can be as consequential as model architecture, and that careful experimental design is essential for drawing reliable conclusions about generalist robot policy performance.