SaPaVe: Vision-Language-Action Framework

• SaPaVe is a unified vision-language-action framework that integrates semantic active perception and robust manipulation by decoupling camera and body actions.
• It employs a geometry-aware conditioning module with a diffusion transformer to fuse 3D spatial data and visual-language features, ensuring viewpoint invariance.
• The framework’s two-stage, data-efficient training protocol and rigorous evaluations set new benchmarks for active robotic manipulation in complex, dynamic environments.

SaPaVe is an end-to-end framework for vision-language-action (VLA) learning in robotics, designed to address unified active perception and manipulation in complex, dynamic scenes. It achieves viewpoint-invariant execution and semantic-driven scene exploration by explicitly decoupling camera actions from manipulation actions and introducing a geometry-aware conditioning module. SaPaVe follows a bottom-up, data-efficient training protocol and is supported by large-scale curated datasets and rigorous simulation and real-world evaluations, setting significant benchmarks for active robotic manipulation beyond fixed-view settings (Liu et al., 12 Mar 2026).

1. Architectural Components and Data Flow

SaPaVe integrates distinct submodules that facilitate learning and joint inference of semantic active perception and robust manipulation:

• Backbone Vision–LLM (VLM): The model employs Eagle-2 (SigLIP-2 paired with SmolLM2), with parameters frozen except for a LoRA “Camera Adapter,” which enables efficient fine-tuning for semantic camera control without retraining the main VLM body.
• Universal Spatial Encoder: The MapAnything feed-forward encoder maps arbitrary 3D geometry inputs—including camera intrinsics/extrinsics, depth maps, and ray directions—into per-patch spatial tokens.
• Diffusion Transformer (DiT) Policy: Conditioned on the fusion of VLM and spatial tokens, the DiT denoises a $T$-step latent vector to predict action chunks for both perception and manipulation.
• Decoupled Action Heads: Final-stage latent representations are split, producing:
  • Camera action vectors $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$ (pitch, yaw)
  • Manipulation action vectors $$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$ (dual-arm & hands)
• Fusion Strategy: VLM tokens ($\phi_{\mathrm{vlm}}$) are passed through the LoRA adapter; spatial tokens $F_{\mathrm{spatial}}$ are obtained via

$$F_{\mathrm{spatial}} = \mathrm{LayerNorm}\bigl(F_{\mathrm{rgb}} + \mathrm{Proj}(F_{\mathrm{geo}})\bigr)$$

Fused representation:

$$\phi_{\mathrm{fused}} = \phi_{\mathrm{vlm}} + \beta \cdot \mathrm{Linear}(F_{\mathrm{spatial}})$$

These fused tokens serve as key/value in every DiT cross-attention block.

This design enables rapid, coordinated active perception (camera control) and manipulation, while isolating the instability that might arise from entangled action spaces.

2. Two-Stage Bottom-Up Training and Losses

SaPaVe's training strategy enables efficient specialization and robust transfer via the following procedures:

Stage 1: Semantic Active Perception Alignment

• Trained on ActiveViewPose-200K (200k image-language-camera movement samples)
• Parameters updated: LoRA Camera Adapter and Camera Action Decoder; Manipulation head and spatial encoder frozen
• Objective: Minimize camera movement prediction error

$$\mathcal{L}_{\mathrm{stage1}} = \frac{1}{T} \sum_{t=1}^T \|\hat{A}_{\mathrm{head}}^t - A_{\mathrm{head}}^{*t}\|_2^2$$

Stage 2: Active Manipulation Fine-tuning

• Data: Mixed batch (ActiveViewPose-200K for sustained camera skill, ActiveManip-Bench and real-world teleop for manipulation/interaction)
• Parameters: All decoders unfrozen; LoRA adapter remains frozen
• Loss:

$$\mathcal{L}_{\mathrm{stage2}} = \lambda_{\mathrm{head}} \mathcal{L}_{\mathrm{head}} + \lambda_{\mathrm{other}} \mathcal{L}_{\mathrm{other}}$$

with

$$\mathcal{L}_{\mathrm{head}} = \sum_{t=1}^T \|\hat{A}_{\mathrm{head}}^t - A_{\mathrm{head}}^{*t}\|_2^2,\quad \mathcal{L}_{\mathrm{other}} = \sum_{t=1}^T \|\hat{A}_{\mathrm{body}}^t - A_{\mathrm{body}}^{*t}\|_2^2$$

and $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$0, $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$1 in practice.

Empirically, ablation studies demonstrate that omitting either stage reduces average success rates significantly.

3. 3D Geometry-Aware Conditioning

The geometry-aware module injects scene structure priors, enabling robust execution across dynamic viewpoints:

• Inputs: Depth map $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$2, camera pose $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$3, intrinsics/extrinsics, ray directions $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$4
• Feature Extraction:
  • $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$5 via frozen DINOv2 (image features)
  • $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$6 via shallow ConvNet on $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$7, MLP on pose parameters
• Fusion: As above, spatial and semantic features are fused via normalized sum and linear projection, used in cross-attention at all DiT layers:

$$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$8

No explicit geometric loss is optimized; robustness emerges from the denoising policy conditioned on fused 3D and semantic information.

4. Datasets and Evaluation Benchmarks

ActiveViewPose-200K:

• 200,000 samples of $$\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$$9, where $$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$0 is a 224×224 suboptimal-view RGB image, $$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$1 is a VLM-augmented instruction, $$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$2 is the ground-truth camera movement (pitch/yaw).
• Assembly via Objaverse assets, Infinigen-generated scenes, thousands of templates, and GPT-4o-based prompt augmentation.

ActiveManip-Bench:

• Based on NVIDIA Isaac Sim, Unitree G1 robot, Inspire hands, and 2-DoF active head
• 100 objects × 20 scenes, 12 granular tasks (atomic, composite, long-horizon)
• Three “visual complexity” initializations: Unoccluded, Occluded, Out-of-View target
• Metric: Binary success rate (geometric thresholding on position, orientation, joint, or fluid metrics)

Dataset/Benchmark	Content	Role in Pipeline
ActiveViewPose-200K	200k (image, language, camera movement)	Stage 1 perception; auxiliary in Stage 2
ActiveManip-Bench	task episodes across 100 objects/20 scenes	Manipulation learning & evaluation

This layered benchmarking enables rigorous evaluation of both viewpoint-agnostic perception and long-horizon manipulation across varying scene complexity.

5. Experimental Results and Ablation Analysis

SaPaVe establishes state-of-the-art results across active perception and manipulation settings. The following table summarizes core findings:

a) Semantic active perception on ActiveViewPose-200K

Model	Val	Test1	Test2	Avg.
Qwen2.5-VL-72B	63.9	65.1	58.0	62.3
Multi-SpatialMLLM	72.8	74.3	63.6	70.2
Gemini-2.5-Pro	73.3	76.5	68.2	72.7
Ours (Stage 1 only)	85.5	89.1	78.3	84.3

b) Active manipulation on ActiveManip-Bench (Simulation)

Setup	Unocc. P.a.P	Occ. P.a.P	OoV P.a.P	Unocc. A.M.	Occ. A.M.	OoV A.M.	Avg.
Fixed Camera	74	46	11	52	27	7	36.2
Fixed + Wrist cameras	83	62	28	66	51	24	52.3
Active head + Wrist cams	86	75	70	74	68	66	73.2
Ours (Active head only)	85	78	72	76	70	68	74.8

c) Real-world teleop fine-tuning

Method	Occl. P.a.P	OoV P.a.P	Occl. A.M.	OoV A.M.	Avg.
π₀	55	45	45	35	45.0
GR00T-N1	60	55	50	50	53.8
Ours	90	85	85	80	85.0

d) Ablation (4 real-world tasks, avg. success)

Ablation	Avg.
– w/o Stage 1	53.8
– w/o Stage 2	66.3
– w/o Decoupled Heads	71.3
– w/o Camera Adapter	73.8
– w/o Uni. Spatial KI	68.8

“Ours” refers to the complete SaPaVe pipeline. Strong performance on occluded and out-of-view scenarios highlights the efficacy of decoupled perception-action learning and geometry-aware policy conditioning.

6. Test-Time Execution Protocol

At inference, SaPaVe alternates perception and manipulation steps, leveraging decoupled control to maintain system stability:

$$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$4 The decoupled architecture allows rapid reorientation (Δ_head) without destabilizing the ongoing manipulation trajectory (Δ_body). A plausible implication is that explicit separation of perception and manipulation control spaces is essential for generalization in dynamic, partially observed environments.

7. Summary and Impact

SaPaVe demonstrates that “human-like” active manipulation in robotics requires (1) explicit specialization in continuous semantic camera control, (2) joint fine-tuning with diverse, hybrid data, and (3) persistent integration of 3D geometry priors. The framework achieves up to 31.25% higher success rates in real-world manipulation tasks compared to GR00T N1 and $$\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$$3, especially in occluded or out-of-view settings (Liu et al., 12 Mar 2026). The contributions, including decoupled action modeling, large-scale active perception data, and geometry-aware policy design, inform future research in VLA robotics under realistic multimodal scene complexity.