When would Vision-Proprioception Policies Fail in Robotic Manipulation?

Abstract: Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.

• The paper identifies that joint vision-proprioception policies exhibit a 15.8% decrease in performance during motion transitions.
• It uses temporally controlled intervention experiments and MDP analysis to show that proprioceptive signals dominate and suppress visual cues.
• The proposed GAP algorithm adaptively modulates proprioceptive gradients, significantly improving success rates in both simulated and real robotic tasks.

Failure Modes and Generalization in Vision-Proprioception Robotic Manipulation Policies

Modality Temporality and Policy Suppression Mechanisms

The paper "When would Vision-Proprioception Policies Fail in Robotic Manipulation?" (2602.12032) investigates the integration of visual and proprioceptive modalities in manipulation policies, focusing on their temporal dynamics and optimization-induced failure modes. A primary finding is that joint vision-proprioception policies can exhibit reduced generalization compared to vision-only policies, particularly during task sub-phases characterized by motion transitions. This is highlighted by the 15.8% performance decrease observed for vision-proprioception policies relative to vision-only baselines in core manipulation tasks.

Empirical analysis via temporally controlled intervention experiments reveals that policies using both modalities degrade when switching from vision-only to vision-proprioception during motion-transition phases, which coincide with critical target localization requirements. Conversely, in motion-consistent phases (steady movement), vision-proprioception policies maintain parity with vision-only approaches. This temporal disparity suggests a suppressed role for the vision modality when the policy is optimized in presence of concise, high-gradient-providing proprioceptive signals.

Figure 1: The generalization of vision-proprioception policies, with vision-proprioception policies performing substantially worse than vision-only policies during motion transitions.

Theoretical analysis grounded in MDP and behavior cloning dynamics demonstrates that vision-proprioception policies, under standard joint-learning architectures and modality concatenation, preferentially minimize loss via proprioceptive channels. Proprioceptive signals, such as gripper pose and opening degree, directly correlate with robot actions and exhibit pixel-level variation, prompting the optimization to attenuate vision chunk gradients. This causally suppresses the vision modality, leading to poor generalization when object positions and task configurations deviate from those seen during training.

Gradient Adjustment with Phase-Guidance (GAP): Algorithm and Implementation

To address this modality suppression, the authors introduce the Gradient Adjustment with Phase-guidance (GAP) algorithm. GAP uses change-point detection on proprioception trajectories to segment manipulation phases, defines motion-transition intervals probabilistically via temporal networks (e.g., LSTM), and adaptively modulates proprioceptive gradient magnitudes during policy learning. By attenuating the proprioceptive backward gradient proportionally to the estimated transition phase indicator $\rho$, GAP aims to force the vision-proprioception policy to leverage visual cues during motion transitions without sacrificing precision in motion-consistent intervals.

Figure 2: Pipeline of the GAP algorithm, highlighting motion representation, phase estimation, and gradient modulation tied to transition indicators.

This mechanism is implemented within both conventional policy architectures (MLP, diffusion, transformer-based) and Vision-Language-Action models, enabling dynamic intra-policy collaboration. During policy optimization, the proprioceptive chunk $\omega_s$ receives a reduced update ($${\omega}_s^{j+1} = {\omega}_s^j - \lambda \cdot (1-\rho) \cdot \eta \nabla {\omega}_s^j \mathcal{L}_{BC}({\omega}_s^j)$$) during motion transitions, as detected by LSTM phase indicators.

Experimental Scope and Numerical Results

Extensive evaluations span Meta-World and RoboSuite simulations, as well as real-world tasks on single and dual-arm robotic setups. Manipulation task diversity ranges from object assembly, bin-picking, and threading to dual-arm handover and pouring, covering both precision- and robustness-sensitive domains.

Figure 3: Visualization of real-world manipulation tasks with both one-arm and dual-arm robot setups.

GAP-equipped vision-proprioception policies consistently outperform all baselines, including vision-only, naive multi-modal concatenation, MS-Bot stage-guided fusion, auxiliary loss, and random mask approaches. Across tasks and modalities, GAP yields average success rate improvements, notably:

• Assembly: 94.2% (GAP) vs. 82.6% (vision-only) in Meta-World
• Threading: 53.0% (GAP) vs. 43.6% (vision-only) in RoboSuite
• Hand-over (dual-arm real): 18/20 (GAP) vs. 15/20 (vision-only)

Moreover, GAP substantially enhances out-of-distribution robustness, with vision-proprioception policies under GAP exceeding vision-only policies (e.g., assembly task OOD: 88% vs. 78%).

Figure 4: Intervention experiment showing enhanced utilization of vision modality within GAP-equipped vision-proprioception policies.

Compatibility with transformer-based VLA models (Octo-VP) is demonstrated: GAP yields a 17% average improvement relative to vision-only fine-tuned models, countering previously reported detrimental effects from additional proprioception.

Policy fusion strategy ablations show GAP universally unlocks policy performance regardless of concatenation, summation, or FiLM-based fusion, outperforming vision-only policies in all tested approaches.

Figure 5: Visualization of broad manipulation tasks in simulated evaluation environments.

Loss curve visualizations and linear-probing analysis confirm that GAP achieves slower initial loss reduction but superior final convergence, with learned visual representations exhibiting improved quality and OOD generalization.

Figure 6: Visualization demonstrating GAP-induced loss curve behavior with ultimate lower convergence.

Practical and Theoretical Implications

The study provides decisive evidence that naive vision-proprioception policy fusion can impair generalization by suppressing vision modality during critical motion transitions. The GAP algorithm offers a solution by dynamically calibrating gradient contributions based on task phase, enabling robust information integration and improved transferability. This insight is relevant for the design of embodied manipulation policies across simulation and real-world deployments, and has strong implications for future model architectures targeting domain adaptation, continual learning, or multi-task generalization.

The theoretical perspective on optimization dominance and modality temporality further lays the groundwork for modalities-aware gradient control techniques in multimodal RL, imitation learning, and vision-language-action modeling. Addressing cross-embodiment scenarios and further integrating phase-aware learning in large transformer-based architectures are promising research directions.

Conclusion

This paper rigorously characterizes the failure modes of vision-proprioception policies in robotic manipulation arising from temporal suppression of vision by proprioception. The GAP algorithm, via motion-transition phase-guided gradient adjustment, demonstrably resolves this issue across diverse policy styles, tasks, and environments, producing robust, generalizable manipulation policies. The results highlight the necessity of fine-grained, temporally adaptive modality fusion for advanced multimodal robotic control.