SSI-Policy: RGB Vision-Language Manipulation

• SSI-Policy is an RGB-only, robot-agnostic framework that decouples perception and control using a Structured Scene Interface combining depth, object layout, and motion trajectories.
• The framework leverages large-scale, action-free video pretraining for perception and few-shot behavioral cloning for control, enabling robust performance in real-world tasks.
• Ablation studies on the LIBERO benchmark show that integrating geometric and motion cues improves overall success rates by over 14 percentage points compared to prior methods.

Searching arXiv for the target paper and closely related context. SSI-Policy is a modular framework for vision-language robotic manipulation built around a Structured Scene Interface (SSI), an RGB-only, robot-agnostic intermediate representation that jointly encodes monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories. The framework is designed for low-data robotic manipulation by decoupling perception from control: the perception stack is trainable on large, action-free video corpora, while the downstream policy is learned from very few demonstrations. In the reported evaluation, SSI-Policy is studied on the LIBERO benchmark under a 10-demonstration few-shot protocol and on 13 real-world tasks spanning spatial reasoning, cross-embodiment transfer, deformable object handling, contact-rich operations, multi-stage sequences, and pick-and-place (Wang et al., 25 Jun 2026).

1. Problem setting and design rationale

Real-world robotic manipulation demands spatial grounding, task-aware reasoning, and precise control. The framework is motivated by the claim that prior methods often trade off scalable task-level reasoning and explicit physical structure: video-based approaches can drift geometrically over long horizons, 3D approaches often require depth sensing, and many flow or trajectory interfaces emphasize motion without an explicit RGB-only geometric representation. SSI-Policy is introduced to address this regime with a unified interface that preserves geometric and motion cues while remaining trainable from RGB video alone (Wang et al., 25 Jun 2026).

A central design choice is the separation of perception and control. SSI is robot-agnostic and trainable from action-free video, whereas the control policy is trained with behavioral cloning from task demonstrations. This arrangement is intended for the low-data regime, where acquiring large quantities of robot interaction data is difficult but video corpora are comparatively abundant. A plausible implication is that SSI-Policy treats perception pretraining and policy learning as distinct statistical problems rather than forcing both burdens onto the same small demonstration set.

2. Structured Scene Interface

At each timestep $t$, SSI is composed of three complementary signals. The first is a monocular depth feature encoding $D_t$. Given a temporal window of RGB frames

$I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$

from $V$ camera views, depth features

$$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$

are extracted by passing each $I_\tau^v$ through a pretrained monocular depth backbone such as DepthAnythingV2 and taking the penultimate feature map of channel-depth $C_d$, for example $128$. Rather than regressing metric depth, $D_t$ captures relative geometric cues sufficient for contact reasoning.

The second component is a language-grounded object layout $B_t$. The instruction $D_t$0 is encoded with a text encoder such as BERT into $D_t$1, after which an off-the-shelf grounding detector, GroundingDINO, proposes bounding boxes $D_t$2 keyed to $D_t$3. These boxes are rasterized into a dense salience map

$D_t$4

with

$D_t$5

The third component is a set of instruction-conditioned 2D motion trajectories $D_t$6. Combining the RGB frame $D_t$7, the instruction embedding $D_t$8, and the layout map $D_t$9, the method samples $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$0 start points $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$1 and forecasts $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$2 future pixel positions

$I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$3

via a Transformer-based motion predictor. Half of the $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$4 starts are drawn uniformly, and half are drawn from the high-salience regions of $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$5. The reported ablations state that geometric and motion cues provide complementary benefits within the shared interface, which directly bears on the choice to combine $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$6, $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$7, and $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$8 rather than using any one of them in isolation (Wang et al., 25 Jun 2026).

3. Fusion, tokenization, and action planning

The three SSI components are fused into a unified representation by first encoding each per-view feature tile—depth $I_t = \{ I_{t-t_o+1}^{1:V}, \ldots, I_t^{1:V} \}$9, layout $V$0, and optionally raw RGB $V$1—with modality-specific 2D CNNs into 128-channel grids of size $V$2. These are concatenated along channels and projected via a small MLP to produce spatial embeddings

$V$3

Each spatial embedding is flattened into tokens of length $V$4, a learnable $V$5 token is prepended, and the flattened trajectory tokens $V$6 projected to 128-D$V$7 are appended to form a single sequence. The full token sequence is enriched with positional embeddings for time, view, and spatial location, together with modality-type embeddings, and then processed by a 7-layer, 8-head Transformer of hidden size 128.

The output $V$8 tokens from each timestep are concatenated with proprioceptive features $V$9 to form the structured scene vector

$$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$0

Once $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$1 is computed, a Diffusion Action Planner (DAP) consumes $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$2 raw RGB $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$3 proprioception $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$4 to generate a horizon-$$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$5 action sequence

$$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$6

Actions are modeled via a conditional denoising diffusion process. At diffusion timestep $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$7, let $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$8 denote the noisy version of the ground-truth $$D_t \in \mathbb{R}^{t_o\times V\times n_h\times n_w\times C_d}$$9, with $I_\tau^v$0. The learning objective is

$I_\tau^v$1

At test time, a pure-noise sequence is iteratively denoised into a coherent action plan. The DAP U-Net resembles the “Diffusion Policy” architecture and embeds spatial grids of $I_\tau^v$2 tokens formed from concatenated RGB, depth, and layout features (Wang et al., 25 Jun 2026).

4. Learning decomposition and the few-shot regime

Perception modules are trained independently on large collections of human or robot videos lacking action labels. The monocular depth module uses a self-supervised or pre-trained depth head and requires no further loss. The language-grounded object locator is an off-the-shelf detector trained on open-vocabulary detection data. The motion predictor is trained with a reconstruction loss on future trajectories,

$I_\tau^v$3

using only video and text.

Because SSI is robot-agnostic and requires no action labels, only the DAP is trained with behavioral cloning from $I_\tau^v$4 demonstrations per task, minimizing

$I_\tau^v$5

The paper attributes the resulting few-shot performance to this clean decoupling and reports that only $I_\tau^v$6 demonstrations per task are needed. This suggests that SSI-Policy is not merely a policy architecture but also a training decomposition: representation learning is sourced from action-free video, while control learning is confined to the scarce demonstration channel (Wang et al., 25 Jun 2026).

5. Benchmark performance and ablation behavior

Evaluation on LIBERO uses four suites—Spatial, Object, Goal, and Long—each with 10 tasks and 50 demonstrations per task. The reported few-shot protocol uses 10 demonstrations per task, 20 rollouts per seed, and three random seeds. Under this setting, SSI-Policy improves over ATM-DP, identified as the strongest prior 10-demo baseline in the comparison, by 14.75 percentage points in average success rate (Wang et al., 25 Jun 2026).

Suite	ATM-DP (10-demo)	SSI-Policy
Spatial	79.00	83.50
Object	81.00	95.00
Goal	58.67	82.50
Long	44.00	60.67
Average	65.67	80.42

With 50 demonstrations, SSI-Policy reaches 91.25% overall success, ranking second among RGB-only methods without external data, and the abstract further states that it remains competitive with 50-demo methods that leverage large-scale external pretraining. The evaluation therefore places the framework simultaneously in a few-shot comparison and in a broader RGB-only comparison regime.

The ablation studies are reported as follows: Geometry-only (“DepthHelps”) yields 63.15% average success; motion-only (“ATM”) 63.42%; Motion+Depth reaches 76.92%; Motion+Layout 77.09%; and the full SSI-Policy reaches 80.42%. Removing the hybrid sampling scheme in the motion predictor lowers performance to 78.46%. The explicit conclusion drawn in the paper is that geometry and motion cues are complementary, and the hybrid sampling result underscores the importance of instruction-aware trajectory initialization. A common simplification is to treat geometric grounding and motion forecasting as interchangeable supervisory signals; these ablations argue against that view.

6. Real-world validation, transfer, and failure modes

Real-world validation spans 13 manipulation tasks across spatial reasoning, cross-embodiment transfer, deformable object handling, contact-rich operations, multi-stage sequences, and pick-and-place. On six spatial reasoning tasks, SSI-Policy achieves 80% average success, versus 43.3% for a diffusion-only baseline. In robot-to-robot transfer, evaluated zero-shot on new arms, SSI-Policy outperforms ATM-based methods by approximately 10–20 points and suffers less degradation. When SSI is pretrained on 100 human-hand videos and fine-tuned with 10 robot demonstrations, the reported success rates rise from 45% to 70% on pot-to-stove and from 40% to 80% on broom-sweep tasks. On five especially challenging real-world tasks, including sponge handling, drawer closing, and wiping, a single multi-task SSI-Policy achieves 54% average success, compared with 29% for per-task diffusion baselines (Wang et al., 25 Jun 2026).

Across simulation and hardware, the paper attributes common failure modes primarily to upstream perception errors: mis-localized or missing detections under occlusion, jitter in monocular depth, and grasp misalignments on deformables. The stated future directions are more robust detectors, temporal consistency, and tactile feedback. Within the framework’s own logic, these failure modes are significant because they identify the main residual bottleneck after decoupling perception from control: if the structured scene interface is corrupted, the downstream diffusion planner inherits that error rather than correcting it.