---
title: 'SaPaVe: Vision-Language-Action Framework'
url: https://www.emergentmind.com/topics/sapave
type: topic
---

# SaPaVe: Vision-Language-Action Framework

SaPaVe is an end-to-end framework for vision-language-action (VLA) learning in robotics, designed to address unified active perception and manipulation in complex, dynamic scenes. It achieves viewpoint-invariant execution and semantic-driven scene exploration by explicitly decoupling camera actions from manipulation actions and introducing a geometry-aware conditioning module. SaPaVe follows a bottom-up, data-efficient training protocol and is supported by large-scale curated datasets and rigorous simulation and real-world evaluations, setting significant benchmarks for active robotic manipulation beyond fixed-view settings [2603.12193].

## 1. Architectural Components and Data Flow

SaPaVe integrates distinct submodules that facilitate learning and joint inference of semantic active perception and robust manipulation:

- **Backbone Vision–Language Model (VLM):** The model employs Eagle-2 (SigLIP-2 paired with SmolLM2), with parameters frozen except for a LoRA “Camera Adapter,” which enables efficient fine-tuning for semantic camera control without retraining the main VLM body.
- **Universal Spatial Encoder:** The MapAnything feed-forward encoder maps arbitrary 3D geometry inputs—including camera intrinsics/extrinsics, depth maps, and ray directions—into per-patch spatial tokens.
- **Diffusion Transformer (DiT) Policy:** Conditioned on the fusion of VLM and spatial tokens, the DiT denoises a $T$-step latent vector to predict action chunks for both perception and manipulation.
- **Decoupled Action Heads:** Final-stage latent representations are split, producing:
  - Camera action vectors $\hat{A}_{\mathrm{head}} \in \mathbb{R}^{T \times 2}$ (pitch, yaw)
  - Manipulation action vectors $\hat{A}_{\mathrm{body}} \in \mathbb{R}^{T\times 26}$ (dual-arm & hands)
- **Fusion Strategy:** VLM tokens ($\phi_{\mathrm{vlm}}$) are passed through the LoRA adapter; spatial tokens $F_{\mathrm{spatial}}$ are obtained via
  $$
  F_{\mathrm{spatial}} = \mathrm{LayerNorm}\bigl(F_{\mathrm{rgb}} + \mathrm{Proj}(F_{\mathrm{geo}})\bigr)
  $$
  Fused representation:
  $$
  \phi_{\mathrm{fused}} = \phi_{\mathrm{vlm}} + \beta \cdot \mathrm{Linear}(F_{\mathrm{spatial}})
  $$
  These fused tokens serve as key/value in every DiT cross-attention block.

This design enables rapid, coordinated active perception (camera control) and manipulation, while isolating the instability that might arise from entangled action spaces.

## 2. Two-Stage Bottom-Up Training and Losses

SaPaVe's training strategy enables efficient specialization and robust transfer via the following procedures:

**Stage 1:** Semantic Active Perception Alignment
- Trained on ActiveViewPose-200K (200k image-language-camera movement samples)
- Parameters updated: LoRA Camera Adapter and Camera Action Decoder; Manipulation head and spatial encoder frozen
- Objective: Minimize camera movement prediction error
  $$
  \mathcal{L}_{\mathrm{stage1}} = \frac{1}{T} \sum_{t=1}^T \|\hat{A}_{\mathrm{head}}^t - A_{\mathrm{head}}^{*t}\|_2^2
  $$

**Stage 2:** Active Manipulation Fine-tuning
- Data: Mixed batch (ActiveViewPose-200K for sustained camera skill, ActiveManip-Bench and real-world teleop for manipulation/interaction)
- Parameters: All decoders unfrozen; LoRA adapter remains frozen
- Loss:
  $$
  \mathcal{L}_{\mathrm{stage2}} = \lambda_{\mathrm{head}} \mathcal{L}_{\mathrm{head}} + \lambda_{\mathrm{other}} \mathcal{L}_{\mathrm{other}}
  $$
  with
  $$
  \mathcal{L}_{\mathrm{head}} = \sum_{t=1}^T \|\hat{A}_{\mathrm{head}}^t - A_{\mathrm{head}}^{*t}\|_2^2,\quad
  \mathcal{L}_{\mathrm{other}} = \sum_{t=1}^T \|\hat{A}_{\mathrm{body}}^t - A_{\mathrm{body}}^{*t}\|_2^2
  $$
  and $\lambda_{\mathrm{head}}=1.0$, $\lambda_{\mathrm{other}}=10.0$ in practice.

Empirically, ablation studies demonstrate that omitting either stage reduces average success rates significantly.

## 3. 3D Geometry-Aware Conditioning

The geometry-aware module injects scene structure priors, enabling robust execution across dynamic viewpoints:
- **Inputs:** Depth map $D$, camera pose $P$, intrinsics/extrinsics, ray directions $R$
- **Feature Extraction:**
  - $F_{\mathrm{rgb}}$ via frozen DINOv2 (image features)
  - $F_{\mathrm{geo}}$ via shallow ConvNet on $(D, R)$, MLP on pose parameters
- **Fusion:** As above, spatial and semantic features are fused via normalized sum and linear projection, used in cross-attention at all DiT layers:
  $$
  \phi_{\mathrm{fused}} = \phi_{\mathrm{vlm}} + \beta\,\mathrm{Linear}(F_{\mathrm{spatial}})
  $$
No explicit geometric loss is optimized; robustness emerges from the denoising policy conditioned on fused 3D and semantic information.

## 4. Datasets and Evaluation Benchmarks

**ActiveViewPose-200K:**
- 200,000 samples of $(I, L, A^*_{\mathrm{head}})$, where $I$ is a 224×224 suboptimal-view RGB image, $L$ is a VLM-augmented instruction, $A^*_{\mathrm{head}}$ is the ground-truth camera movement (pitch/yaw).
- Assembly via Objaverse assets, Infinigen-generated scenes, thousands of templates, and GPT-4o-based prompt augmentation.

**ActiveManip-Bench:**
- Based on NVIDIA Isaac Sim, Unitree G1 robot, Inspire hands, and 2-DoF active head
- 100 objects × 20 scenes, 12 granular tasks (atomic, composite, long-horizon)
- Three “visual complexity” initializations: Unoccluded, Occluded, Out-of-View target
- Metric: Binary success rate (geometric thresholding on position, orientation, joint, or fluid metrics)

| Dataset/Benchmark    | Content                                    | Role in Pipeline                            |
|----------------------|--------------------------------------------|----------------------------------------------|
| ActiveViewPose-200K  | 200k (image, language, camera movement)    | Stage 1 perception; auxiliary in Stage 2     |
| ActiveManip-Bench    | task episodes across 100 objects/20 scenes | Manipulation learning & evaluation           |

This layered benchmarking enables rigorous evaluation of both viewpoint-agnostic perception and long-horizon manipulation across varying scene complexity.

## 5. Experimental Results and Ablation Analysis

SaPaVe establishes state-of-the-art results across active perception and manipulation settings. The following table summarizes core findings:

### a) Semantic active perception on ActiveViewPose-200K

| Model                   | Val   | Test1 | Test2 | Avg.  |
|-------------------------|-------|-------|-------|-------|
| Qwen2.5-VL-72B          | 63.9  | 65.1  | 58.0  | 62.3  |
| Multi-SpatialMLLM       | 72.8  | 74.3  | 63.6  | 70.2  |
| Gemini-2.5-Pro          | 73.3  | 76.5  | 68.2  | 72.7  |
| Ours (Stage 1 only)     | 85.5  | 89.1  | 78.3  | 84.3  |

### b) Active manipulation on ActiveManip-Bench (Simulation)

| Setup                    | Unocc. P.a.P | Occ. P.a.P | OoV P.a.P | Unocc. A.M. | Occ. A.M. | OoV A.M. | Avg.  |
|--------------------------|--------------|------------|-----------|-------------|-----------|----------|-------|
| Fixed Camera             | 74           | 46         | 11        | 52          | 27        | 7        | 36.2  |
| Fixed + Wrist cameras    | 83           | 62         | 28        | 66          | 51        | 24       | 52.3  |
| Active head + Wrist cams | 86           | 75         | 70        | 74          | 68        | 66       | 73.2  |
| Ours (Active head only)  | 85           | 78         | 72        | 76          | 70        | 68       | 74.8  |

### c) Real-world teleop fine-tuning

| Method      | Occl. P.a.P | OoV P.a.P | Occl. A.M. | OoV A.M. | Avg. |
|-------------|-------------|-----------|------------|----------|------|
| π₀          | 55          | 45        | 45         | 35       | 45.0 |
| GR00T-N1    | 60          | 55        | 50         | 50       | 53.8 |
| Ours        | 90          | 85        | 85         | 80       | 85.0 |

### d) Ablation (4 real-world tasks, avg. success)

| Ablation               | Avg.  |
|------------------------|-------|
| – w/o Stage 1          | 53.8  |
| – w/o Stage 2          | 66.3  |
| – w/o Decoupled Heads  | 71.3  |
| – w/o Camera Adapter   | 73.8  |
| – w/o Uni. Spatial KI  | 68.8  |

*“Ours” refers to the complete SaPaVe pipeline. Strong performance on occluded and out-of-view scenarios highlights the efficacy of decoupled perception-action learning and geometry-aware policy conditioning.*

## 6. Test-Time Execution Protocol

At inference, SaPaVe alternates perception and manipulation steps, leveraging decoupled control to maintain system stability:

```python
Input: language instruction L
Initialize: noised action chunk A^τ, t=0
loop until task done:
  1. Capture RGB image I_t and (optional) geometry G_t
  2. φ_vlm ← Eagle-2(I_t, L) + LoRA adapter
  3. F_spatial ← MapAnything(G_t)  # encoder on depth, rays, poses
  4. φ_fused ← φ_vlm + β·Linear(F_spatial)
  5. For τ from max_noise down to 0:
       ε̂ = DiT_denoiser(A_t^τ, φ_fused)
       A_t^{τ−1} = denoise_step(A_t^τ, ε̂, τ)
     end
  6. Decode:
       Δ_head = CameraDecoder(A_t^0)
       Δ_body = BodyDecoder(A_t^0)
  7. Execute head movement Δ_head on the robot
  8. Execute manipulation action Δ_body on the arms/hands
  9. t ← t+1, capture new I_t, G_t
end
```
The decoupled architecture allows rapid reorientation (Δ_head) without destabilizing the ongoing manipulation trajectory (Δ_body). *A plausible implication is that explicit separation of perception and manipulation control spaces is essential for generalization in dynamic, partially observed environments.*

## 7. Summary and Impact

SaPaVe demonstrates that “human-like” active manipulation in robotics requires (1) explicit specialization in continuous semantic camera control, (2) joint fine-tuning with diverse, hybrid data, and (3) persistent integration of 3D geometry priors. The framework achieves up to 31.25% higher success rates in real-world manipulation tasks compared to GR00T N1 and $π_0$, especially in occluded or out-of-view settings [2603.12193]. The contributions, including decoupled action modeling, large-scale active perception data, and geometry-aware policy design, inform future research in VLA robotics under realistic multimodal scene complexity.

Source: https://www.emergentmind.com/topics/sapave