VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models

Abstract: Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, but their real-world evaluation remains limited by a lack of accessible, reproducible, and consistent benchmarks. Simulation benchmarks fail to capture real-world complexity, while existing real-world benchmarks often require expensive hardware, centralized evaluation, or are limited in task diversity. We introduce VLA-REPLICA, a low-cost, easily reproducible real-world benchmark for evaluating VLA models. Built from off-the-shelf components, our system can be quickly assembled and replicated across laboratories, providing a consistent environment for policy evaluation anywhere in the world. VLA-REPLICA includes a diverse suite of manipulation tasks and a small-scale demonstration dataset for target-domain adaptation, with real-world evaluation protocols for both in-distribution and out-of-distribution settings. Experiments with imitation learning and state-of-the-art VLA models reveal model strengths and limitations, while consistent results across independently constructed setups demonstrate the reproducibility of our benchmark.

• The paper introduces VLA-REPLICA, a low-cost, reproducible benchmark for real-world evaluation of vision-language-action models.
• It standardizes evaluations using off-the-shelf hardware and calibrated setups across diverse in-distribution and out-of-distribution tasks.
• Empirical results show that fine-tuned VLA models outperform baselines, though sequential memory tracking remains a significant challenge.

VLA-REPLICA: A Reproducible Benchmark for Real-World Vision-Language-Action Model Evaluation

Motivation and Design Criteria

Vision-language-action (VLA) models have demonstrated robust performance in robotic manipulation tasks prompted by natural language inputs. However, evaluation in realistic environments remains problematic due to the sim-to-real gap, high hardware costs, limited task diversity, and the lack of reproducible, distributed benchmarks. Existing simulation platforms (e.g., Meta-World, LIBERO, RoboCasa) enable standardized evaluation but fail to encapsulate the complexity of real-world scenarios. Prior real-world benchmarks either restrict focus to basic tasks (e.g., REPLAB, SceneReplica), require costly equipment (FurnitureBench, ManipulationNet), or depend on centralized evaluation limiting transparency and laboratory autonomy.

VLA-REPLICA addresses these deficiencies by providing a low-cost ($\sim$ \$1050), easily assembled platform using off-the-shelf components and standardized protocols for real-world evaluation of VLA models. Its explicit goals are reproducibility, accessibility, and broad task coverage.

Benchmark Platform and Standardization

The hardware configuration centers on the SO-101 follower arm, Intel RealSense D455 RGB-D camera (top-view), and a commodity RGB webcam (wrist view), all installed in a 32-inch cubic photography light box to ensure consistent illumination. Calibration is enforced via AprilTag reference and video overlay matching to standardize workspace, camera poses, and robot alignment in every setup, both for demonstration collection and policy evaluation. Any SO-101 arm, regardless of manufacturer variance, is calibrated to a universal action space using encoder offsets and linear scaling, ensuring comparability across hardware instances.

Figure 1: Illustration of the hardware components and manipulation workspace, along with the diverse task suite in VLA-REPLICA.

Figure 2: Methodology for reproducible setup, demonstrating task space calibration, object placement matching, and unified evaluation.

Task Suite, Dataset, and Evaluation Protocols

VLA-REPLICA consists of ten manipulation tasks encompassing pick-and-place, object interaction, and memory-centric actions (e.g., counting repetitions), covering a wide array of behaviors from stacking blocks to folding towels and opening an oven. For each task, 50 expert demonstrations are collected via teleoperation, yielding a 500-sample dataset for fine-tuning and adaptation. Tasks are evaluated under both in-distribution (ID) and out-of-distribution (OOD) conditions—ID tasks probe fit to target environment after adaptation; OOD tasks systematically introduce shifts in object attributes, configurations, and requirements to test generalization.

Test scenes are strictly reproduced using reference images and overlay tools, controlling for spatial and attribute deviations from training. The evaluation suite comprises 90 standardized test scenes.

Figure 3: Examples of demonstration data and reference test scene images used for reproducible policy evaluation.

Empirical Benchmarking and Analysis

A spectrum of models, including imitation learning baselines (ACT, Diffusion Transformers with/without flow-matching objective) and state-of-the-art VLA models (SmolVLA, X-VLA, $\pi_0$, $\pi_{0.5}$), were benchmarked:

• All models were trained or fine-tuned on the same 500 demonstrations for 40K steps.
• Fine-tuned VLA policies overwhelmingly outperform baselines trained from scratch in both ID and OOD distributions.
• Among all models, $\pi_{0.5}$ achieves the highest average success rates on both ID (0.54) and OOD (0.35) tasks, with strong performance on pick-and-place and deformable/manipulation tasks.
• Success rates for memory-counting tasks (e.g., repeated pouring or lifting) are low across all models, indicating a deficiency in temporal action tracking, persistent memory, or sequence execution fidelity.

  Figure 4: Comparative evaluation showing model success rates for ID/OOD tasks and reproducibility across laboratory setups.

Reproducibility Outcomes

A key validation is reproducibility: evaluation on an independently constructed replica setup produced almost identical success rates to the original environment for selected ID and OOD tasks, demonstrating the robustness of calibration and overlay protocols. Policy outputs were transformed to the physical action space of each SO-101 instance, ensuring universal interpretability.

Figure 5: Physical workspace view and reference calibration—reproducibility is achieved by precise object, arm, and camera placement.

Task Suite Reference and Composition

Each task is defined by precise object arrangements, success criteria, and both ID and OOD variants to enforce diversity and avoid overfitting to particular configurations.

Figure 6: Reference images for Task 1 ("Put Bread on Plate") illustrating both ID and OOD scene variations.

Figure 7: Reference images for Task 2 ("Put Bowl on Coaster") with multiple color and placement configurations.

Figure 8: Reference images for Task 3 ("Stack Blocks") for color and positional generalization.

Implications and Future Directions

VLA-REPLICA represents a practical tool for robust assessment of vision-language-action policies in real-world, reproducible, and distributed setups. By eliminating centralized evaluation, providing affordable hardware, and enforcing standardized scene composition, it unlocks rapid iteration and transparent benchmarking for both academia and industry.

Empirical results evidence that current VLA architectures, even when pretrained, still underperform on tasks involving sequential action counts and complex temporal dependencies—a key limitation for real-world generalist robotics. The reproducibility protocol sets a new precedent for rigorous, geographically distributed evaluation that can accommodate future extensions in robot embodiment diversity, scene complexity, and large-scale multi-modal datasets.

Further research will need to address memory-centric modeling, scalable expansion to non-tabletop domains, and incorporation of richer sensory modalities to enhance generalization. The benchmark’s modular and open design invites community contributions and comparative studies across labs, stimulating progress in the field.

Conclusion

VLA-REPLICA enables consistent, low-cost real-world evaluation of vision-language-action models under standardized, reproducible conditions. Fine-tuning on the provided demonstration dataset yields strong adaptation for pretrained models, but persistent challenges remain for memory-driven manipulation. Evaluation results are reproducible across hardware instances, indicating benchmark robustness. VLA-REPLICA provides a foundation for broad, distributed benchmarking that can accelerate methodological advances and deployment readiness for general-purpose robotic systems (2605.20774).