- The paper introduces StereoPolicy, which fuses stereo visual features using a Transformer to enable precise spatial reasoning in robotic manipulation.
- It achieves significantly higher success rates in both simulation and real-world tasks compared to conventional monocular and explicit 3D reconstruction baselines.
- The approach minimizes calibration needs and enhances robustness to occlusions and reflective surfaces while incurring only a modest inference latency increase.
StereoPolicy: Advancing Robotic Manipulation via Direct Stereo Perception
Motivation and Context
Robotic visuomotor policy learning has historically depended on monocular visual inputs processed by large pretrained 2D vision encoders for downstream action generation. While such pipelines attain substantial performance in uncluttered environments, they are fundamentally limited by a lack of reliable depth estimation and spatial reasoning—critical for manipulation in geometrically complex or occluded settings. Existing strategies to incorporate spatial cues typically involve explicit 3D reconstruction (through RGB-D or point cloud representations), but these suffer in challenging real-world scenarios due to their reliance on precise calibration and their susceptibility to noise, reflective or transparent surfaces, and artifacts in depth sensing.
StereoPolicy Architecture
StereoPolicy proposes an alternative: leveraging synchronized stereo image pairs directly, processed by pretrained 2D vision backbones (e.g., ResNet18, DINOv2, CLIP) and fused with a cross-attention Stereo Transformer to produce stereo-aware embeddings without the need for explicit 3D reconstruction or specialized calibration. The key architectural components are:
- Stereo Feature Extraction: Each image in a stereo pair is encoded independently with shared-weights 2D vision encoders to generate single-view features. For external views, DINOv2 embeddings are concatenated to strengthen semantic priors especially in ambiguous regions.
- Stereo Transformer Fusion: The feature maps from each pair are fused in a Stereo Transformer using alternating self- and cross-attention layers, augmented by 2D RoPE positional encoding. This promotes geometric correspondence and implicit spatial reasoning.
- Integration with Policy Backbones: The fused stereo features are concatenated with low-dimensional proprioceptive state inputs and passed into policy heads—either a diffusion policy (STEREOPOLICY-DP) or as a drop-in replacement for monocular vision modules in large, pretrained vision-language-action (VLA) policies (STEREOPOLICY-VLA).
Experimental Evaluation
Benchmarks and Tasks
StereoPolicy was evaluated across a comprehensive suite of simulated and real-world tasks, spanning standard benchmarks (RoboMimic, OmniGibson, RoboCasa) and challenging real-world scenarios involving both desktop and mobile bimanual manipulation. Control baselines included monocular RGB, RGB-D, multi-view RGB, point-cloud, 3DDA, and DP3-based architectures.
Numerical Results and Findings
- Consistent Outperformance: Across all real-world table-top manipulation tasks, STEREOPOLICY-DP achieved an average success rate of 59%, compared to 41–45% for RGB/RGB-D/multi-view RGB and <30% for point cloud-based policies. On particularly challenging tasks (such as glass/steel cup hang), explicit 3D methods failed entirely whereas StereoPolicy maintained nonzero success rates.
- Simulation Task Dominance: On simulation benchmarks, STEREOPOLICY-DP outperformed all baselines, particularly under low-data regimes and for tasks demanding fine-grained geometric reasoning (e.g., ToolHang, PourWater). For instance, average success rate on OMNIGibson tasks was 822.8% (with perfect scores on some tasks), compared to 593.4–744.3% for RGB/RGB-D baselines.
- VLA Model Integration: Adding StereoPolicy as a visual input module for pretrained VLA models (e.g., Pi0.5, GR00T-N1.5) yielded consistent improvements across all data regimes, increasing average success rate up to 4% in high-data and 3% in low-data scenarios—validating the hypothesis that stereo encoders can supplement the spatial limitations of VLM pretraining, even when original weights are trained on monocular data.
- Robustness to Hardware and View Changes: Best results were obtained when the stereo baseline is approximately 10% of the camera-object distance (r∈[0.09,0.13]), offering practical guidance for camera placement. The performance gains from stereo were most pronounced for front- and off-axis views with high geometric ambiguity and occlusions, while gains were reduced for side views.
- Backbone and Component Ablation: Larger vision-LLMs (e.g., OPENCLIP-B/16, SIGLIP-SO400M/14) significantly improved sample efficiency and absolute success rates over smaller CNNs, especially when coupled with external-view DINOv2 features. Omitting the Stereo Transformer led to marked degradation in performance, underlining the necessity of effective feature fusion for stereo modalities.
- Efficiency: Inference latency increased modestly (1.12x) over RGB-only policies, predominantly due to stereo feature encoding and fusion—but was competitive with or superior to explicit 3D-reconstruction baselines.
Implications for Robotic Perception and Policy Learning
Practical Impact
StereoPolicy demonstrates that policy learning frameworks can directly profit from stereo vision—a robust, easily deployable, and scalable modality—without the brittle dependence on explicit depth or point cloud reconstructions. This finding holds particular promise for real-world deployments, where transparent, reflective, or occluded objects invalidate the assumptions underlying commodity depth sensors and classical 3D fusion pipelines. The approach is hardware-agnostic with respect to most commercial stereo rigs, reduces the need for precise calibration, and is computationally tractable for real-time inference.
Theoretical Implications
The empirical superiority of implicit stereo feature fusion over explicit geometric representations highlights the effectiveness of cross-attention architectures in extracting actionable spatial correspondence from 2D input modalities. The data also suggest that pretraining on large-scale, monocular semantic datasets can be successfully leveraged for stereo-based downstream control when paired with appropriate fusion mechanisms.
Open Questions and Future Directions
StereoPolicy’s gains on reflective and transparent objects, though statistically significant against baselines, leave room for improvement in absolute terms. The approach is sensitive to adverse illumination and extreme viewing geometries, prompting further investigation into adaptive augmentation, temporal fusion across viewpoints, and improved stereo rig calibration or active lighting. Scaling StereoPolicy to broader embodiments and large-scale, in-the-wild robot datasets (e.g., DROID) could extend its success to more diverse manipulation and navigation domains. Integration with multi-modal fusion networks or 3D-aware transformers, and longitudinal adaptation for long-horizon tasks, are promising areas for future research.
Conclusion
StereoPolicy establishes stereo vision as an effective and scalable pathway for geometric reasoning in robotic manipulation, filling the gap between scalable 2D pretrained perception and the need for reliable 3D spatial awareness. The cross-attention-based fusion architecture empowers policies to surpass established monocular, RGB-D, and point cloud-based baselines both in laboratory and real-world robot deployments. Its seamless integration with both diffusion-based and large-scale VLA backbones, combined with practical design guidelines for stereo setup, positions StereoPolicy as a robust, general-purpose advancement for contemporary visuomotor robot learning frameworks.
