SpatialActor: Disentangled Spatial Manipulation

• SpatialActor is a robotic manipulation framework that decouples semantic and geometric features to overcome limitations of point-based and RGB-D methods.
• It integrates a Semantic-guided Geometric Module and a Spatial Transformer to fuse multi-modal data and provide precise 2D-3D spatial encoding.
• Empirical results in simulation and real-world settings show significant improvements in success rates and noise robustness compared to prior SOTA techniques.

SpatialActor is the framework introduced in "SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulation" (Shi et al., 12 Nov 2025). In its strict sense, the term denotes a robotic manipulation architecture that explicitly decouples semantics and geometry in order to address three stated deficiencies of prior point-based and RGB-D image-based methods: loss of fine-grained semantics under sparse point sampling, brittleness caused by entangled semantics and geometry under noisy depth, and insufficient use of low-level spatial cues needed for precise interaction (Shi et al., 12 Nov 2025). In a broader inferred sense, the term also aligns with a recurring actor-centric spatial reasoning pattern in adjacent literatures on human-object interaction detection and video action localization, where an actor or agent is treated as the organizing reference for scene reasoning rather than as one object among many (Xu et al., 2022, Pan et al., 2020).

1. Problem formulation and representational stance

SpatialActor is framed against two existing families of manipulation methods. Point-based 3D methods explicitly encode geometry, but their sparse sampling tends to discard fine-grained semantic information and they are expensive to annotate or pretrain at scale. Image-based RGB-D methods benefit from pretrained 2D backbones, but fuse semantics and geometry in a shared 2D feature space; the paper argues that this entanglement makes the representation brittle because depth noise, lighting variation, and reflections corrupt geometry and also interfere with semantic features (Shi et al., 12 Nov 2025).

The architectural response is a disentangled representation with three components. Semantic features are derived from RGB and language. Geometric features are derived from depth. Within the geometric pathway, the framework further separates robust high-level geometry from low-level spatial cues relevant to precise contact, alignment, and insertion. This separation is the central design move: it reduces cross-modal interference while preserving explicit spatial structure for action generation (Shi et al., 12 Nov 2025).

The paper implements this design through two named modules. The Semantic-guided Geometric Module (SGM) adaptively fuses geometry from raw depth and geometry priors from a pretrained depth-estimation expert. The Spatial Transformer (SPT) injects explicit spatial position encoding into transformer tokens so that reasoning is anchored in robot-centric 3D position rather than only in appearance-level similarity. The resulting system is evaluated across simulation and real-world settings spanning 50+ tasks (Shi et al., 12 Nov 2025).

2. Semantic-guided Geometric Module

The SGM is motivated by a specific complementarity. Raw depth contains fine structure but is noisy. A frozen pretrained depth expert derived from RGB produces a more stable but coarser geometric prior. SpatialActor combines these sources rather than choosing between them (Shi et al., 12 Nov 2025).

For each view $v$, the model obtains semantic features from RGB, raw geometric features from depth, and robust geometric priors from the expert:

$$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$

Fusion is performed by a learned gate:

$$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$

$$F_{\text{fuse-geo}}^v = G^v \odot F_{\text{geo}}^v + \bigl(1-G^v\bigr)\odot \hat{F}_{\text{geo}}^v.$$

This gate determines how much to trust raw depth and how much to trust the expert prior at each feature location or channel. The intended effect is dual: preserve fine local geometry when raw depth is reliable, and fall back to semantic-guided geometry when raw depth is corrupted. The paper identifies this mechanism as a primary reason for the method’s robustness under degraded depth sensing (Shi et al., 12 Nov 2025).

The representational logic is explicitly asymmetric. RGB and language are treated as reliable carriers of semantics, while geometry is handled as a signal that must be stabilized without erasing detail. This suggests that SpatialActor does not regard multimodal fusion as simple concatenation; instead, it constrains fusion by preserving a distinction between what a region is and where it is in 3D space. That implication is consistent with the stated goal of robust manipulation under noisy conditions (Shi et al., 12 Nov 2025).

3. Spatial Transformer and action parameterization

The Spatial Transformer addresses the paper’s second stated gap: even robust geometry is insufficient if low-level spatial relations are not explicitly represented. Manipulation depends on exact contact points, relative layout, and 2D-3D correspondence. SpatialActor therefore constructs spatial tokens with explicit robot-centric 3D position (Shi et al., 12 Nov 2025).

For each view $v$, fused spatial features are represented as

$H^v \in \mathbb{R}^{N_v \times D},$

and proprioception $P$ is projected and added:

$\widetilde{H}^v = H^v + \mathrm{MLP}(P).$

A pixel $(x',y')$ with depth $d = D^v(x',y')$ is lifted into 3D by camera projection:

$$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$0

where $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$1 is the intrinsic matrix and $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$2 is the extrinsic matrix. The token is then augmented by a 3D positional encoding inspired by RoPE and Fourier features. With axis-wise frequencies

$$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$3

the position-encoded token is

$$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$4

The attention structure is two-stage. First, view-level interaction applies self-attention and an FFN within each view, refining intra-view structure. Second, scene-level interaction concatenates tokens from all views together with language features $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$5, followed by another self-attention and FFN stage to fuse cross-view information, language grounding, and scene-level spatial context (Shi et al., 12 Nov 2025).

Prediction proceeds through a lightweight decoder, ConvexUp, which outputs per-view 2D heatmaps. The 2D target location is obtained via $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$6, then lifted into 3D using the camera model. The final action is parameterized as

$$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$7

with 3D translation, Euler-angle rotation, and gripper state. Supervision comprises three losses: cross-entropy on per-view 2D heatmaps for translation, cross-entropy on discretized Euler angles, and binary classification loss for the gripper state. The paper presents this as a more stable alternative to direct unconstrained regression of the entire action vector (Shi et al., 12 Nov 2025).

4. Evaluation, robustness, and empirical profile

The principal simulation benchmark is RLBench. The setup uses a Franka arm in tabletop scenarios, four fixed RGB-D cameras, image resolution $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$8, action space defined by translation, rotation, and gripper open/close, and OMPL-generated trajectories. Training uses 18 tasks, 249 variations, 100 expert demonstrations per task, evaluation on 25 unseen episodes, approximately 40k iterations, a cosine schedule with 2000-step warmup, batch size 192 on 8 GPUs, CLIP as visual-language encoder, and Depth Anything v2 as geometry expert (Shi et al., 12 Nov 2025).

Setting	Result	Note
RLBench average success	87.4%	2.3 average rank
RLBench vs. RVT-2	+6.0%	previous SOTA reference
Few-shot on 19 novel tasks	79.2%	RVT-2: 46.9%
Real-world overall	63%	RVT-2: 43%

On RLBench, SpatialActor achieves 87.4% average success with 2.3 average rank, which the paper states is about 6.0% better than the previous SOTA RVT-2. The gains are strongest on precision-sensitive tasks, notably Insert Peg: 93.3%, a +53.3% improvement over RVT-2, and Sort Shape: 73.3%, a +38.3% improvement. These task-level results are consistent with the claim that the representation is particularly effective when exact geometry matters (Shi et al., 12 Nov 2025).

Robustness is probed by injecting Gaussian noise into reconstructed point clouds. The paper defines three regimes: Light with 20% points corrupted and standard deviation $$F_{\text{geo}}^v = \mathcal{E}_{\text{raw}}(D^v), \qquad \hat{F}_{\text{geo}}^v = \mathcal{E}_{\text{expert}}(I^v).$$9, Middle with 50% points corrupted and standard deviation $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$0, and Heavy with 80% points corrupted and standard deviation $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$1. Average success improvements over RVT-2 are +13.9%, +16.9%, and +19.4% respectively. On Insert Peg, the advantage under noise is especially pronounced: +88.0% in Light, +78.6% in Middle, and +61.3% in Heavy (Shi et al., 12 Nov 2025).

The pretrained model is also adapted to 19 novel tasks using only 10 demonstrations per task, where SpatialActor achieves 79.2% compared with 46.9% for RVT-2. On ColosseumBench, evaluated over 20 tasks under spatial perturbations, the reported results are 57.4% under No-Vars, 59.2% under MO-Size, 62.0% under RO-Size, and 54.2% under Cam Pose. These are described as consistently above the baselines, indicating robustness under environmental spatial variation (Shi et al., 12 Nov 2025).

Real-world evaluation uses a WidowX-250 arm with an Intel RealSense D435i camera in a fixed front-facing setup, images downsampled from $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$2 to $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$3, calibrated camera-to-robot alignment, 8 real-world tasks, 15 total variants, and 25 demonstrations per task. Overall performance is 63% for SpatialActor versus 43% for RVT-2. Selected examples include Pick Glue to Box: 85% vs 50%, Push Button: 90% vs 67%, Place Carrot to Box: 65% vs 30%, Insert Ring onto Cone: 50% vs 20%, and Wipe Table: 80% vs 50% (Shi et al., 12 Nov 2025).

5. Relation to actor-centric spatial reasoning in adjacent literatures

Although the exact term SpatialActor is formalized in robotics by (Shi et al., 12 Nov 2025), adjacent literatures provide closely related actor-centric formulations. In image-based HOI detection, "Effective Actor-centric Human-object Interaction Detection" constructs a binary actor mask $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$4, concatenates it with RGB to form a 4-channel RGBM input, predicts actor and object interaction areas with separate branches, and composes final HOI scores through center-point indexing. The design goal is to reduce ambiguity in crowded scenes by conditioning global reasoning on a selected actor (Xu et al., 2022).

In weakly supervised spatiotemporal action localization, "Guess Where? Actor-Supervision for Spatiotemporal Action Localization" uses actor proposals obtained from an SSD detector with an InceptionV2 backbone and links them through a Siamese similarity-based tracker. Action localization is then learned with an actor-based attention mechanism from video-level labels only. Here the actor tube functions as the structured hypothesis space through which spatial and temporal evidence are organized (Escorcia et al., 2018).

In video action localization, ACAR-Net introduces the Actor-Context-Actor relation $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$5, implemented through the High-order Relation Reasoning Operator (HR$$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$6O) and the Actor-Context Feature Bank (ACFB). The method reasons about two actors through shared context regions rather than only through direct pairwise relations, and reports gains on AVA, UCF101-24, and AVA-Kinetics. This is a more explicit higher-order formalization of actor-centric spatial context (Pan et al., 2020).

A related AVA action-detection architecture preserves actor RoI layout as a $$G^v = \sigma\Big(\mathrm{MLP}\big(\mathrm{Concat}(\hat{F}_{\text{geo}}^v, F_{\text{geo}}^v)\big)\Big),$$7 spatial actor tensor and applies two cross-attention blocks: one for spatial context from the slow pathway and one for short-range temporal context from the fast pathway. The paper’s core claim is that temporal evidence should remain non-aggregated until relation modeling, rather than being collapsed too early (Calderó et al., 2021).

Actor identification further extends the same organizing principle. ASAD, or Actor-identified Spatiotemporal Action Detection, requires prediction of an actor’s spatiotemporal boundary, unique actor identity, and action labels. Its A-AVA benchmark and metrics—[email protected], IDF1, MT, ML, ID Switches, and [email protected]—formalize the idea that “who is doing what” is not fully captured by conventional spatiotemporal action detection (Yang et al., 2022).

Taken together, these works suggest a broader interpretive reading: SpatialActor can denote not only a particular robotic manipulation framework, but also a cross-domain modeling stance in which a spatially situated actor or agent is the anchor for disambiguating context, interactions, and action labels. That interpretation is inferred from the convergent design patterns across the cited works.

6. Limitations, failure modes, and significance

SpatialActor’s reported failures remain instructive. In simulation, failure cases arise from instruction misunderstanding, long-horizon breakdowns, and semantic confusion among similar objects. In real-world settings, failures are attributed to pose precision limits, instruction misunderstanding, and distractor clutter in the background. The paper suggests several remedies: better instruction parsing with LLMs, episodic memory or belief tracking for long-horizon tasks, uncertainty-aware pose refinement, and attention-based filtering for clutter (Shi et al., 12 Nov 2025).

The ablation study supports the claim that the final performance is not attributable to a single component. On 18 tasks, the baseline scores 81.4 under no noise and 57.0 under heavy noise; adding semantic-geometric decoupling yields 85.1 and 68.7; adding SGM yields 86.4 and 73.9; and adding SPT yields 87.4 and 76.4. The contribution therefore lies in the full stack of semantic-geometric disentanglement, gated geometric fusion, and spatially explicit token interaction (Shi et al., 12 Nov 2025).

In the narrower technical sense, SpatialActor is a robotic manipulation system that combines RGB-language semantics, noisy depth, depth-expert priors, and explicit 2D-3D spatial encoding to generate accurate manipulation actions under substantial sensing corruption (Shi et al., 12 Nov 2025). In the broader inferred sense suggested by related HOI and action-localization research, it exemplifies a general shift away from undifferentiated scene encoding toward actor- or agent-centered spatial reasoning, where geometry, context, and identity are modeled as structured relations rather than collapsed into a single latent representation (Xu et al., 2022, Pan et al., 2020).