MSACT: Multistage Spatial Alignment for Stable Low-Latency Fine Manipulation

Abstract: Real-world fine manipulation, particularly in bimanual manipulation, typically requires low-latency control and stable visual localization, while collecting large-scale data is costly and limited demonstrations may lead to localization drift. Existing approaches make different trade-offs: action-chunking policies such as ACT enable low-latency execution and data efficiency but rely on dense visual features without explicit spatial consistency, generative methods such as Diffusion Policy improve expressiveness but can incur iterative sampling latency, vision-language-action and voxel-based methods enhance generalization and geometric grounding but require higher computational cost and system complexity. We introduce a multistage spatial attention module that extracts stable 2D attention points and jointly predicts future attention sequences with a temporal alignment loss. Built upon ACT with a pretrained ResNet visual prior, a multistage attention module extracts task-relevant 2D attention points as a local spatial modality for action prediction. To maintain consistent object tracking, we introduce a self-supervised objective that aligns predicted attention sequences with visual features from future frames, suppressing drift without keypoint annotations and improving stability of the vision-to-action mapping under limited data. Experiments on simulated and real-world fine manipulation tasks, conducted on the ALOHA bimanual platform, evaluate task success, attention drift, inference latency, and robustness to visual disturbances. Results indicate improvements in localization stability and task performance while maintaining low-latency inference under the tested conditions.

• The paper introduces a multistage spatial attention module that extracts task-relevant 2D points to ensure stable localization and reduce visual drift.
• It integrates self-supervised temporal alignment with an Action Chunking Transformer, achieving low-latency, data-efficient closed-loop control.
• Empirical results demonstrate significantly higher success rates and robustness on both simulated and real-world bimanual tasks under visual disturbances.

MSACT: Multistage Spatial Alignment for Stable Low-Latency Fine Manipulation

Introduction

MSACT proposes a multistage spatial alignment approach for robotic fine manipulation that directly addresses the core challenges of bimanual tasks: achieving low-latency closed-loop control, spatial localization stability, and data efficiency in the presence of limited demonstrations. While previous methods such as ACT, Diffusion Policy, and various vision–language–action (VLA) and voxel-based architectures have improved expressive policy learning or spatial generalization, each carries distinct trade-offs in computational overhead, robustness, or latency. MSACT introduces a lightweight and principled approach by augmenting ACT with a structured 2D attention-point modality, extracted via a novel multistage spatial attention (MSA) module, and enforces temporal alignment to suppress visual drift during deployment.

Methodology

Multistage Spatial Attention (MSA) Module

At the core of MSACT is the MSA module, which extracts a compact set of task-relevant 2D attention points from multiple RGB camera views at each timestep. Unlike single-scale or one-stage attention extraction pipelines, the MSA module utilizes a three-stage convolutional hierarchy, generating multiscale query and key features before dot-product attention and spatial normalization. The resulting attention maps are fused and regularized to maintain both coarse and fine spatial structure. A differentiable soft-argmax with absolute positional embeddings produces $6 \times 2$ spatial coordinates per view, maintaining geometric fidelity and allowing interpretability.

Figure 1: Overview of the MSACT architecture, illustrating the integration of multistage spatial attention into the ACT core pipeline.

Figure 2: The architecture of the multistage spatial attention module, showcasing the multiscale feature fusion and attention point extraction.

Self-supervised Temporal Alignment

MSACT leverages a self-supervised temporal alignment loss to enforce temporal smoothness and consistency in the predicted attention-point sequences. The model jointly predicts K-step action and attention-point sequences, and aligns predicted attention points with those re-extracted from future ground-truth frames using an $\ell_1$ loss. This suppresses drift, even without explicit keypoint annotations, and ensures the spatial interface between perception and control remains robust across interaction stages and visual disturbances.

End-to-end Integrability and Policy Structure

MSACT is built atop the Action Chunking Transformer backbone, inheriting its low-latency execution and data-efficient chunk-wise policy learning paradigm. Attention-point tokens from both Top and Front cameras are concatenated with image and robot state features, producing a hybrid representation that combines explicit spatial geometry and rich visual context. The resulting tokens are ingested by the sequence model for simultaneous future action and spatial trajectory prediction, facilitated by the conditional VAE framework and temporal ensembling for smoothing closed-loop execution.

Experimental Results

Task Suite and Baselines

Evaluation is conducted on both simulation and challenging real-world bimanual tasks using the ALOHA platform. Real-world tasks include network cable detachment, velcro threading, tea bag insertion, and match box opening—each requiring force modulation, precise mid-air coordination, and robustness to occlusions and appearance variation. Four representative baselines are compared against: ACT, Diffusion Policy, SmolVLA, and $\pi0.5$ (VLA). All methods train or adapt on 50 demonstrations per task under identical settings.

Figure 3: The four real-world bimanual manipulation tasks used for evaluation, each composed of multiple fine-grained subtasks.

Quantitative Performance and Robustness

On simulated tasks (Cube Transfer, Bimanual Insertion), MSACT achieves higher stage and full-task success rates compared to ACT and a single-stage spatial attention ablation. In real-world settings, MSACT demonstrates significantly higher completion and subtask success rates than all baselines in contact-rich and mid-air alignment phases. The overall real-world task success rate for MSACT reaches 53.00% (99% CI: [46.58–59.33]), more than double that of ACT (23.25%) and substantially outperforming SmolVLA (15.25%) and $\pi0.5$ (13.00%).

MSACT operates at 45.40±5 ms per inference, on par with ACT and far below VLA and Diffusion Policy inference latencies, thus fulfilling real-time control requirements.

Spatial Localization and Visual Interpretability

Visualization of attention points and ResNet image feature activations across task phases reveals that MSACT yields more consistent and coherent spatial localization of task-relevant entities even during contacts, handovers, and state transitions. Unlike single-stage or dense-feature approaches, attention points remain stably fixed on manipulated objects and key interaction sites, preventing drift towards background or irrelevant distractors.

Figure 4: Comparison of predicted attention points for MSACT and single-stage ablations, with stable, task-relevant point tracking in MSACT.

Figure 5: Stage-wise visualization of attention points and averaged ResNet feature activations across all four real-world tasks, highlighting stable localization and intensity alignment with manipulation phases.

Robustness to Visual Disturbances

Performance analyses under colored illumination, adversarial distractors, and dynamic hand intrusions illustrate that, while pretrained dense features (e.g., ResNet outputs) can show transient shifts, the MSACT attention points are resilient, providing stable spatial grounding and maintaining reliable manipulation performance under all tested adverse conditions.

Figure 6: Stability of predicted attention points under visual disturbances including unseen distractors, lighting changes, and human interference.

Attention-point Modality Ablation

Normalized cross-attention analysis in the Transformer decoder reveals that the attention-point modality exhibits peak attention intensity during critical spatial manipulation phases, such as object contact, precise insertion, and coordination. This dynamic allocation reflects the value of structured geometric priors over raw high-dimensional features and supports the hypothesis that explicit spatial tokens facilitate more stable and interpretable action decoding, especially in limited-data scenarios.

Implications and Future Directions

These results substantiate the benefit of incorporating explicit, multistage spatial representations within end-to-end visuomotor policies. Structured 2D attention-point modalities provide a powerful geometric interface between visual perception and control policy, enhancing physical grounding, drift suppression, and trajectory interpretability, without the latency or complexity overhead of full 3D or VLA models.

Practically, this architecture enables scalable deployment of robust low-latency fine manipulation policies on affordable hardware, reducing reliance on depth sensing and comprehensive pretraining. Theoretically, it supports a move toward hybrid spatial-attentional representations and self-supervision as an effective regularizer for sequential robot policies.

Future research should explore integrating language-conditioned modulation of spatial attention into this framework, further bridging the gap between data-efficient structured manipulation, open-world generalization, and semantic multi-tasking. Extensions to vision–language–action transformers may capitalize on the interpretability and efficiency of attention-point tokens, yielding more robust and explainable embodied intelligence for general-purpose robotics.

Conclusion

MSACT presents a theoretically motivated and practically effective approach for attaining stable, low-latency fine manipulation by embedding multistage spatial alignment into action-chunking policies. The strong empirical gains in success rates, spatial robustness, and interpretability across diverse task regimes substantiate the utility of structured spatial representation with temporal alignment. This work suggests a promising direction for scalable and efficient visuomotor control in complex real-world environments.