SSI-Policy: Learning Structured Scene Interfaces for Vision-Language Robotic Manipulation

Abstract: Real-world robotic manipulation demands spatial grounding, task-aware reasoning, and precise control. Learning such capabilities becomes particularly challenging in the low-data regime. Prior methods often trade off scalable task-level reasoning and explicit physical structure: video-based approaches can drift geometrically over long horizons, 3D approaches often require depth sensing, and many flow/trajectory interfaces emphasize motion without an explicit RGB-only geometric representation. We introduce SSI-Policy, a modular framework built around a Structured Scene Interface (SSI) -- a unified, RGB-only intermediate representation that jointly encodes monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories. Critically, SSI is robot-agnostic and trainable from action-free video, decoupling perception from control so that the downstream policy can learn from few demonstrations. On the LIBERO benchmark with only 10 demonstrations per task, SSI-Policy improves over the strongest prior method by nearly 15\% and remains competitive with 50-demo methods that leverage large-scale external pretraining. Ablations show that geometric and motion cues provide complementary benefits within the shared interface. We further validate on 13 real-world tasks spanning spatial reasoning, cross-embodiment transfer, and contact-rich manipulation.

• The paper presents a modular SSI framework that decouples perception from control to achieve high data efficiency in vision-language robotic manipulation.
• The approach leverages monocular geometry, language-grounded layout maps, and instruction-conditioned trajectories to enhance spatial reasoning and task performance.
• Experimental results demonstrate 80%+ few-shot success and robust cross-embodiment transfer, validating SSI as a scalable mid-level interface.

SSI-Policy: Data-Efficient Vision-Language Robotic Manipulation via Structured Scene Interfaces

Introduction and Motivation

The paper "SSI-Policy: Learning Structured Scene Interfaces for Vision-Language Robotic Manipulation" (2606.26800) proposes a modular policy framework that advances data-efficient language-conditioned robot manipulation by constructing a mid-level perceptual interface—the Structured Scene Interface (SSI). Robotic policies require spatial grounding, task-aware reasoning, and precise control, especially under the constraint of few-shot data. Previous approaches either operate directly in pixel space, imposing prohibitive computational requirements and geometric drift, or demand explicit metric depth via RGB-D sensing, which limits scalability and versatility. In contrast, SSI enables learning from action-free RGB videos and decouples perception from control, significantly increasing efficiency and cross-embodiment transferability.

Structured Scene Interface Design

SSI is architected to serve as a unified, robot-agnostic intermediate representation for vision-language manipulation:

• Monocular Geometry: SSI leverages monocular depth feature maps from RGB inputs (Depth Anything V2), extracting relative spatial relationships without requiring metric depth sensors.
• Language-Grounded Layout Maps: Task-relevant regions are highlighted by feeding language instructions into a grounded object detector (Grounding DINO), aggregating bounding box confidences into dense spatial maps.
• Instruction-Conditioned Motion Trajectories: SSI forecasts motion via hybrid sampling—starting pixel trajectories uniformly across the RGB frame, focused on high-confidence layout regions for both object and non-object dynamics.

Each signal is independently trainable from action-free videos, allowing modular pretraining and broad reuse across scenes and embodiments.

Figure 1: SSI-Policy pipeline, transforming RGB observations and language instructions into SSI encoding geometry, layouts, and trajectories for task-agnostic action generation.

Policy Architecture and Diffusion-Based Control

The proposed architecture comprises two modules:

• Perception Composer: Converts sensory observations (RGB, language) into SSI using three pretrained subnets for depth, layout map, and motion trajectory prediction.
• Diffusion Action Planner (DAP): Fuses multi-modal features (RGB, depth, layout, trajectories) and proprioception into a structured scene vector via CNN-MLP-Transformer pipeline. Actions (joint and gripper states) are generated through conditional denoising diffusion, leveraging multimodal conditioning for robust, long-horizon control.

  Figure 2: SSI-Policy framework diagram: Perception Composer extracts structured signals; Diffusion Action Planner fuses modalities for action sequence generation.

  

  Figure 3: Multi-modal feature fusion: modality-specific CNNs encode per-view features, trajectory tokens integrated via Transformer, forming the condition vector for denoising diffusion.

Experimental Results: LIBERO and Real-World Benchmarks

SSI-Policy is benchmarked on LIBERO (four suites: Spatial, Object, Goal, Long) and 13 real-world manipulation tasks. Strong empirical findings include:

• Few-shot Success: With only 10 demonstrations per task, SSI-Policy achieves $80.42\%$ average success rate, surpassing ATM-DP by nearly $+15\%$ and outperforming multiple baselines which require significantly more data or external pretraining.
• Scalability: SSI-Policy remains competitive in the 50-demo regime (average $91.25\%$ success), second only to OFT—demonstrating that modular mid-level interfaces do not impede performance even at scale.
• Spatial Reasoning: SSI-Policy excels on real-world spatial reasoning tasks—e.g., disambiguating spatial references and handling deformable objects—with $80\%$ average success rate, markedly higher than RGB-only diffusion policies.
• Cross-Embodiment Transfer: SSI enables zero-shot and few-shot robot-to-robot and human-to-robot transfer by pretraining motion predictors on diverse action-free videos. Notably, using human-hand data alone achieves $40-45\%$ success, reaching $70-80\%$ when combined with robot demonstrations.

  Figure 4: Overview of the real-world task suite and experimental setup: 6-DoF arm, multi-view RGB, diverse task categories.

  

  Figure 5: Cross-embodiment LIBERO-Spatial results: SSI-Policy demonstrates superior few-shot and zero-shot success rates compared to prior interfaces.

  

  Figure 6: Policy ablation: Comparison of full SSI+RGB policy with SSI-only variant, confirming interface sufficiency for generalized behavior.

  

  Figure 7: Qualitative snapshots of real-world and simulation rollouts, visualizing SSI overlays and policy execution.

Ablation Studies

Component-wise ablation confirms that each SSI signal is necessary, and their combination yields maximal performance. Depth cues improve spatial reasoning; layout maps optimize goal selection; motion trajectories enhance contact-rich manipulation. Removing hybrid trajectory sampling decreases performance, supporting the necessity of instruction-conditioned initialization. SSI-only policy retains $98\%$ performance of the full model, validating the interface as an adequate surrogate for raw observations.

Robustness and Limitations

SSI-Policy demonstrates robustness under challenging conditions: deformable object manipulation, contact-rich operations, and cluttered or ambiguous scenes. Failure cases primarily arise from upstream perception inaccuracies (e.g., detection under occlusion, jitter from monocular estimation), not the interface or policy itself. Potential improvements include temporally consistent perception, stronger detection backbones, and tactile feedback.

Implications and Future Directions

Pragmatically, SSI-Policy enables high-performance, data-efficient manipulation without reliance on depth sensing or massive external datasets, making it deployable across diverse robotic platforms. Theoretically, SSI sets a precedent for decoupling perception and control via structured mid-level representations, facilitating cross-domain transfer, modular pretraining, and scalable robotic learning. The abstraction strategy has immediate implications for multi-agent transfer learning, hybrid human-robot policy distillation, and efficient multi-task generalization.

Future research will focus on temporally stable perception pipelines, physically motivated tactile feedback, and further integration with cognitive generalist agent architectures that leverage embodied large-scale LLMs.

Conclusion

SSI-Policy establishes a structured, RGB-only scene interface that enhances vision-language robotic manipulation efficiency, enables embodiment-agnostic learning, and achieves strong performance in both data-scarce and scalable regimes. The framework's abstraction approach provides robust spatial reasoning, task-aware trajectory generation, and seamless cross-embodiment policy transfer, signifying a substantive advance in modular robotic policy learning (2606.26800).