STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

Abstract: Robotic manipulation critically requires reasoning about future spatial-temporal interactions, yet existing VLA policies and world-model-enhanced policies do not fully model action-relevant spatial-temporal interaction structure. We propose STARRY, a world-model-enhanced action-generation policy that aligns spatial-temporal prediction with action generation. STARRY jointly denoises future spatial-temporal latents and action sequences, and introduces Geometry-Aware Selective Attention Modulation to convert predicted depth and end-effector geometry into token-aligned weights for selective action-attention modulation. On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings. Real-world experiments further improve average success from 42.5% to 70.8% over $π_{0.5}$, demonstrating the effectiveness of action-centric spatial-temporal world modeling for spatial-temporally demanding robotic action generation.

• The paper presents a novel world-model-enhanced policy, STARRY, that combines spatial-temporal prediction with geometry-aware attention to generate precise action sequences.
• It introduces the GASAM mechanism, which uses depth maps and token-level weights to modulate action decisions based on predicted spatial structure.
• Extensive simulation and real-world evaluations demonstrate significant improvements in success rates for complex, contact-rich manipulation tasks.

STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

Motivation and Problem Statement

Robotic manipulation, particularly in the context of language-conditioned tasks, requires robust reasoning about the future spatial-temporal evolution of the interaction scenario. Traditional Vision-Language-Action (VLA) policies, while effective at mapping multimodal perception and language inputs to actions, remain predominantly reactive and lack explicit mechanisms for anticipating future object configurations, spatial constraints, or action outcomes. World-model-enhanced policies have attempted to address this gap through latent video or observation prediction, yet fall short in directly aligning predictive spatial-temporal structure with action generation—especially in contact-rich or geometrically demanding manipulation settings.

The key problem addressed in this paper is the need for models that are simultaneously action-centric and geometry-aware: able to generate action sequences that are tightly coupled to predictions of future scene geometry and spatial-temporal interaction structure, supporting manipulation tasks that demand fine-grained alignment, precise contact, and anticipatory coordination.

STARRY Policy Architecture

STARRY (Spatial-Temporal Action-centric Representation and Reasoning PolicY) is introduced as a world-model-enhanced action-generation policy that tightly integrates spatial-temporal future prediction with the process of action synthesis. Its architecture is characterized by several interacting components:

• Understanding Expert: Extracts semantic grounding from vision-language input, ensuring instruction-conditioned control.
• Spatial-Temporal (ST) World Model: Predicts structured future spatial-temporal latent variables, capturing appearance, motion, and 3D geometry over time, and aligning them with the temporal horizon considered during action generation.
• Geometry Expert & GASAM: Forms the crux of STARRY's geometry-awareness via the Geometry-Aware Selective Attention Modulation (GASAM) mechanism. It predicts depth maps and end-effector trajectories, constructs spatially aligned, token-wise geometry-aware weights, and uses these weights to modulate the action attention mechanism selectively.
• Action Expert: Generates actions by jointly denoising over the predicted future spatial-temporal latents and employing the modulation signal for spatially focused decision-making.

This modular configuration (Figure 1) enables explicit injection of predicted geometric relevance directly into the attention mechanism governing action choices, bridging the gap between perceptual foresight and control.

Figure 1: Overview of the STARRY architecture, integrating spatial-temporal prediction, geometric modulation, and action generation via four key modules.

Formalism and Predictive World Modeling

Instead of direct observation-to-action mapping, STARRY introduces internal variables: future spatial-temporal latent variables and geometry-aware weights. The control policy optimizes a joint generative model:

$$\pi_\theta\left(\mathbf{a}_{t+1:t+H}, \mathbf{z}_{t+1:t+H} \mid \mathbf{o}_t\right)$$

where $\mathbf{z}$ models predictive spatial-temporal structure and $\mathbf{w}$ modulates attention in the action branch based on geometric cues.

Construction of the spatial-temporal input (Figure 2) involves unifying raw RGB, depth, and end-effector trajectory observations across multiple camera views. These are aggregated through camera projection and composition functions into a dense tokenized representation capturing appearance, motion, and underlying geometry. This representation is encoded and diffused into future spatial-temporal latents that are directly shared with the action-generation process.

Figure 2: Multimodal fusion into shared spatial-temporal inputs, combining RGB, depth, and trajectory for comprehensive future prediction.

Geometry-Aware Selective Attention Modulation (GASAM)

GASAM is a decisive innovation introduced in STARRY. Given predicted depth and end-effector position, GASAM computes token-level geometry-aware weights, which reflect the spatial proximity of each visual element to the anticipated manipulator position. These weights (Figure 3) explicitly bias the action-to-video attention in transformer layers:

$$\operatorname{Attn}^{a\leftarrow v}_{\mathrm{GASAM}} = \operatorname{Softmax}\left( \frac{\mathbf{Q}^a (\mathbf{K}^v)^\top}{\sqrt{d}} + \lambda \log(\mathbf{w} + \epsilon) \right)\mathbf{V}^v$$

This augmented attention mechanism assures that action queries prioritize visual tokens corresponding to geometrically critical regions, thus focusing policy capacity on action-relevant, spatially proximate structures.

Figure 3: Visualization of GASAM attention modulation—geometry-aware weights highlight action-relevant visual regions for spatially focused decision-making.

Experimental Evaluation

Simulation: RoboTwin 2.0

STARRY is validated on the RoboTwin 2.0 benchmark, a suite of 50 bimanual manipulation tasks with structured randomization. The model outperforms baselines ($\pi_{0.5}$, X-VLA, Motus, LingBot-VA) both on averaged metrics and, crucially, on spatial-temporally challenging tasks:

• Average success: 93.82% (Clean) / 93.30% (Randomized)
• Highest per-task improvements are observed in tasks demanding precise alignment and temporal coordination, e.g. Hanging Mug, Handover Mic, Turn Switch.

This demonstrates that the advantages of joint spatial-temporal modeling and geometry-aware attention are most apparent in settings requiring precise, anticipatory, and space-aware actions.

Real-World Experiments

On ARX R5 hardware, STARRY achieves strong empirical gains compared to $\pi_{0.5}$ on multi-stage bimanual tasks such as Hand Over Vegetables, Tidy Up Room, and Wash Baby Bottle (Figure 4). Average real-world success improves from 42.5% to 70.8%, with gains especially prominent in multi-step (stage 2) scenarios where spatial-temporal foresight is essential.

Figure 4: Real-world settings for evaluation, illustrating complex two-stage bimanual manipulation tasks tested on ARX R5 hardware.

Qualitative side-by-side comparisons show that STARRY maintains superior alignment, contact precision, and temporal consistency in execution (see reference to qualitative comparison figures in the text).

Predictive Latent Visualization

Analysis of decoded future latents (Figures 6, 7) confirms that STARRY's internal spatial-temporal prediction preserves object layouts, manipulator trends, and depth structure across horizons, indicating that its world model captures meaningful, geometrically structured foresight, rather than trivial visual consistency.

Figure 5: Future spatial-temporal latent rollouts in simulation, retaining scene coherence, object location, and depth structure.

Figure 6: Decoded future latents in physical robot execution, showing consistent RGB-D structure and end-effector motion over time.

Ablation and Component Analysis

Quantitative ablation studies highlight STARRY's architectural strengths:

• Spatial-temporal foresight (joint denoising of latent and action sequences) delivers a significant boost over action-only or appearance-only future prediction.
• GASAM yields up to +10.92% (Randomized) and +9.08% (Clean) absolute improvement over non-modulated policies, with maximal effect when combined with full spatial-temporal prediction.
• Improvements are most pronounced on manipulation tasks sensitive to contact localization, object alignment, and placement, substantiating the complementary value of structured future modeling and geometry-aware attention.

Theoretical and Practical Implications

STARRY demonstrates that aligning structured, geometry-aware world modeling directly with action generation constitutes a substantial improvement in robotic manipulation. By tightly coupling future latent prediction, explicit geometric priors, and selective spatial attention, STARRY produces agents that are not only more reliable in average-case settings, but robustly outperform on tasks with intricate spatial-temporal dependencies or requiring high-fidelity interaction with the environment.

Practically, the approach underscores the importance of unifying multimodal perception, scene geometry, and trajectory prediction when scaling embodied AI to complex, real-world tasks—moving beyond reactive policies toward agents capable of anticipation, planning, and precise interaction.

Theoretically, STARRY opens avenues for:

• Further research into explicit spatial-temporal structure learning as an inductive bias for closed-loop control.
• Exploration of unsupervised or weakly-supervised spatial grounding through geometry-predicted attention in broader settings.
• Extensions to open-vocabulary or multi-agent manipulation, given the strong grounding in both language and geometry.

Conclusion

STARRY sets a new standard for world-model-enhanced manipulation policies by integrating deep spatial-temporal modeling and explicit geometry-aware control. Its empirical results highlight the necessity of action-centric and geometry-attentive world models for advanced robotic manipulation. The approach has implication for both scaling embodied AI systems to greater robustness in natural environments, and for the principled design of spatially aware action policies that generalize to the full complexity of physical interaction tasks.