Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances

Abstract: Robots operating in open, unstructured real-world environments must rely on onboard visual perception while autonomously moving across different locations. Continuous changes in onboard camera viewpoints cause significant visual scale variations in target objects, affecting vision-based motion generation. In this work, we present a stereo multistage spatial attention-based deep predictive learning method for real-time mobile manipulation. The proposed methods extracts task-relevant spatial attention points from stereo images and integrates them with robot states through a hierarchical recurrent architecture for closed-loop action prediction. We evaluate the system on four real-world mobile manipulation tasks using a mobile manipulator, including rigid placement, articulated object manipulation, and deformable object interaction. Experiments under randomized initial positions and visual disturbance conditions demonstrate improved robustness and task success rates compared to representative imitation learning and vision-language-action baselines under identical control settings. The results indicate that structured stereo spatial attention combined with predictive temporal modeling provides an effective solution within the evaluated mobile manipulation scenarios.

• The paper presents a novel stereo multistage spatial attention module paired with hierarchical LSTM to enhance real-time mobile manipulation.
• It achieves an 85% mean success rate across diverse tasks, significantly outperforming monocular and single-stage approaches under severe visual disturbances.
• The system delivers ultra-low latency (~33 ms per step on CPU), making it practical for deployment on resource-constrained mobile robots.

Stereo Multistage Spatial Attention for Robust Real-Time Mobile Manipulation

Introduction

The paper "Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances" (2605.00471) advances robust real-time closed-loop mobile manipulation through a novel stereo multistage spatial attention (MSA) architecture integrated with hierarchical LSTM-based predictive control. The motivation stems from the practical challenges that real-world mobile robots face due to continual viewpoint changes, large apparent scale variations, and frequent visual disturbances—scenarios where existing end-to-end visuomotor and imitation learning baselines struggle to generalize.

The proposed system leverages stereo vision to extract geometrically consistent, task-relevant spatial attention points and fuses these representations temporally via hierarchical LSTMs. A comprehensive empirical evaluation across four diverse manipulation tasks under severe initial condition and visual disturbance variation benchmarks the method against state-of-the-art imitation learning and vision-language-action (VLA) policies, providing insight into both architectural contributions and deployment feasibility.

Figure 1: Examples of the mobile manipulation tasks studied, with blue lines indicating robot trajectories and the onboard camera’s capture of the target object.

Methodology

The system builds on deep predictive learning principles, utilizing stereo vision to overcome viewpoint-induced variations. The main architectural contributions are twofold:

1. Stereo Multistage Spatial Attention (MSA) Module: The MSA module processes left/right stereo image inputs through parallel, weight-shared convolutional pipelines to extract multiple attention points corresponding to salient task-relevant features (e.g., object centroids, grasp affordances). Unlike previous single-stage attention approaches, MSA aggregates information across three feature stages, enabling both fine-grained and coarse attention and significantly improving spatial consistency under visual appearance changes and occlusions.
2. Hierarchical LSTM Motion Predictor: Extracted stereo attention points and proprioceptive robot states are processed by a hierarchical LSTM stack. Three parallel low-level LSTMs encode left attention, right attention, and joint state trajectories. A high-level LSTM integrates their cell states, mediating multimodal temporal dependencies and stabilizing long-horizon prediction for closed-loop action output.

The loss function comprises mean squared prediction error on motor commands, a smoothing prior for continuity, and a temporally bidirectional unsupervised attention-point term, balancing exploration and precision during training.

Figure 2: Overview of the pipeline, including the multistage spatial attention module, motion prediction, temporally bidirectional loss, and hierarchical LSTM design.

Figure 3: Detailed MSA module architecture, contrasting previous single-scale SA with the proposed multistage design and its generated multi-point attention outputs.

The system is implemented for fast CPU inference, enabling real-time deployment on constrained onboard hardware common in mobile robots.

Experimental Results

The evaluation focuses on four distinct tasks (rigid placement, articulated-object manipulation, grasp with pose uncertainty, and deformable object interaction) with randomization over initial robot and object configurations. Each policy is tested for 50 trials per task, and ablations include removing stereo inputs or reverting to single-stage attention. Comparisons are made to ACT, Diffusion Policy, and large VLA baselines ($\pi_0$, SmolVLA), all re-trained or fine-tuned with identical datasets and subject to receding-horizon actuation constraints.

Key empirical findings:

• The proposed stereo MSA + LSTM model achieves a mean success rate of 85.0% across all tasks, significantly outperforming baselines (ACT 46%, Diffusion Policy 28.5%, $\pi_0$ 29%, SmolVLA 12.5%).
• The monocular or single-stage attention ablations result in substantial performance degradation (down to 33-38%), confirming the necessity of both stereo and multistage spatial representations.

  Figure 4: Experimental mobile manipulation setup showcasing the dual-arm platform, stereo cameras, and teleoperation-based data collection pipeline.

  

  Figure 5: Visualization of spatial attention and its predicted evolution over time with the MSA network on representative real-world sequences.

• The model maintains stable attention tracking, even under challenging conditions such as unseen backgrounds, visual distractors, and severe lighting variation, where ResNet-based policies (ACT) exhibit diffused or errant focus and poor generalization.
• Attention point stability analysis demonstrates that multistage fusion constrains attention drift, particularly in long-horizon tasks or when environmental features become occluded.

  Figure 6: Comparison of attention trajectory stability between single-stage SA and multistage MSA; MSA yields less drift and better focus retention.

  

  Figure 7: Examples of attention map evolution and the effect of multistage aggregation in focusing attention across task time.

• Under severe visual disturbance, the stereo MSA system maintains an overall average success rate of 76.8%, while ACT drops to 24.8%. Integrating MSA-derived attention points into ACT boosts its robustness, but still lags the proposed approach, likely due to LSTM’s BPTT facilitating temporally stable attention in contrast to transformer sampling.

  Figure 8: Robustness of attention point extraction under visual distractors, lighting changes, and background perturbations—MSA is markedly more stable than ResNet-based ACT.

• When tested with increased initial object-robot distances (100 cm, 150 cm, unseen at train time), the method generalizes with almost no performance degradation up to 100 cm, unlike transformer or diffusion-based policies.

  Figure 9: Attention tracking and success rates as a function of initial distance, highlighting generalization to unseen settings.

• Analysis via PCA of internal LSTM representations reveals that stereo MSA uniquely encodes structured, object-position-dependent trajectories in feature space, in contrast to less structured, less correlational embeddings for ablations.

  Figure 10: Principal component embedding of hidden LSTM states, with stereo MSA showing smooth, position-aware manifolds.

• Inference latency is extremely low (~33 ms per step on CPU), far below large VLA baselines or diffusion models, supporting practical real-time implementation on resource-limited platforms.

Implications and Future Directions

The presented architecture advances the deployment of end-to-end visuomotor policies in unstructured real-world mobile manipulation by demonstrating that stereo multistage spatial attention, temporally integrated with hierarchical LSTMs, provides significant gains in robustness, sample efficiency, and real-time capability. The superiority on generalization benchmarks, especially under distributional shift (initial pose, unseen distractors, adverse lighting), is notable given the sample efficiency relative to RL and the typically high computational and data demands of transformer/VLA models.

These results suggest several implications:

• Attention Mechanism Design: Multistage attention with explicit supervision (or bidirectional loss) is critical for maintaining task-relevant spatial localization under severe visual uncertainty.
• Sensor Modality Fusion: Stereo perception is central for acquiring geometric invariance and resolving scale/pose ambiguities—critical for closed-loop mobile manipulation.
• Temporal Modeling: RNN-based predictive modeling with attention feedback can outperform transformer-style chunking policies on tasks that demand temporally consistent, real-time operation—particularly important for mobile robots with limited onboard compute.
• Sample Efficiency: The approach achieves high performance with modest expert demonstration data, critical for scalable lifelong learning in robotic applications.

Prospective work should investigate scaling the method to broader task distributions, integrating spatial attention with transformer-based policies for hybrid architectures, and extending stereo attention concepts for non-rigid manipulation and long-horizon, multi-object scenes.

Conclusion

This study delivers a robust, sample-efficient, and real-time mobile manipulation system by synergistically combining stereo multistage spatial attention with hierarchical temporal modeling. The empirical evidence demonstrates substantial practical and theoretical gains over strong modern baselines. These findings lay the groundwork for further research into scalable visuomotor control using advanced attention mechanisms and stereo perception in open-world robotics.