ActiveViewPose-200K: Semantic Camera Control

• ActiveViewPose-200K is a purpose-built dataset with 200K image–language–camera movement triplets that enable precise semantic camera control in robotics.
• It uses a rigorous semi-automated pipeline combining 3D scene synthesis, task-template generation, and language augmentation to define optimal and perturbed camera poses.
• The dataset supports training language-conditioned camera policies and benchmarks models, demonstrating about a 16% improvement in semantic camera control over baseline systems.

ActiveViewPose-200K is a large-scale, purpose-built dataset designed to enable learning of instruction-conditioned, continuous camera movements for semantic active perception, primarily in robotics and embodied AI contexts. Comprising 200,000 image–language–camera movement triplets, it facilitates the development and evaluation of vision-language-action models capable of translating free-form natural language instructions into fine-grained egocentric camera pose adjustments. The dataset is integral to frameworks such as SaPaVe, which decouples camera control from manipulation in the learning of active perception and execution policies (Liu et al., 12 Mar 2026).

1. Data Collection Methodology and Annotation Pipeline

ActiveViewPose-200K is constructed using a rigorous semi-automated pipeline that combines procedural 3D scene synthesis, semantic task templating, and controlled language augmentation. The workflow consists of several stages:

• 3D Asset Curation and Scene Synthesis: Approximately 4,000 filtered meshes sourced from Objaverse—restricted to common indoor object categories (kitchenware, tools, decorations)—undergo topology, stability, and texture validation. These are instantiated into roughly 500 procedurally generated indoor environments using the Infinigen simulator, spanning kitchens (32%), living rooms (18%), dining rooms (29%), and bathrooms (21%).
• Task-Template Generation: A library of 3,000 semantic task templates is constructed, spanning three major prompt modalities:
  • Visual Centering (e.g., commands to center a partially visible object)
  • Spatial Directive (explicit relative cues such as "look left and pick…")
  • Common-Sense (implicit cues such as "pick the mug in the base cabinet")
  • Additional templates include conditional reasoning and container interaction prompts.
• Optimal-and-Initial Viewpoint Pairing: For each scene–object–template instance, a heuristic identifies an “optimal” camera pose that centers the target and minimizes occlusion. This pose is perturbed within ±15° in pitch/yaw to create an “initial” viewpoint. The ground-truth camera movement is defined as the relative head rotation Δ from the initial to the optimal pose.
• Language Instruction Augmentation: Each instruction is rephrased using GPT-4, ensuring linguistic diversity while maintaining equivalent semantics (imperative mood, precise object references, no spurious steps). All generated instructions are manually spot-checked to eliminate hallucinations and misalignments.

2. Dataset Composition and Structure

ActiveViewPose-200K comprises 200,000 samples, each consisting of an image, a natural language instruction, and a sequence of camera movement primitives. Key statistical properties are summarized below:

Attribute	Value/Type	Notes
Scenes	500 procedurally generated indoor environments	Four room categories; room-type proportions
Objects	≈ 4,000 unique household/tool meshes	From Objaverse, validated for quality
Task Templates	3,000 base prompts, GPT-4 augmented to ≈200k	Multiple semantic, spatial, commonsense
Samples	200,000	Each: image, instruction, camera movement
Movement Primitives	4–8 head rotation steps per sample	2-DoF (Δpitch, Δyaw)

Image observations are rendered at H=224, W=224 under varied lighting and clutter. Instructions are UTF-8 strings of ≤40 tokens. Camera movement is represented as a chunk of k relative head rotations, $$a_\text{head}^t = [\Delta \text{pitch}, \Delta \text{yaw}]^\top \in \mathbb{R}^2$$ for $t=1,\ldots,k$ steps.

3. Data Representation and Formal Properties

Each sample is structured as $(I, L, A_\text{head})$ where:

• Image $I \in \mathbb{R}^{224\times224\times3}$
• Language instruction $L$ (≤40 tokens)
• Movement chunk $A_\text{head} = \{a_\text{head}^t\}_{t=1}^k$

Each $a_\text{head}^t$ is a 2-DoF relative camera rotation (Δpitch, Δyaw). For incremental application:

$$R_{t+1} = R(\Delta\text{pitch}, \Delta\text{yaw}) \cdot R_t$$

where $R_{t}$ is the cumulative rotation in $SO(3)$. While the dataset is limited to 2-DoF head rotations, the formalism allows generalization to full 6-DoF camera pose if extended.

The semantic camera control policy is defined as:

$t=1,\ldots,k$0

with a primary training objective of minimizing the mean squared error (MSE) between predicted and ground-truth head rotations over chunk length $t=1,\ldots,k$1:

$t=1,\ldots,k$2

Alternatively, MSE can be computed on total cumulative rotation.

4. Dataset Splits and Evaluation Protocols

ActiveViewPose-200K is partitioned to support robust model development and systematic evaluation:

• Train: 80% (~160,000)
• Validation: 10% (20,000)
• Test1: 5% (10,000) — instructions with explicit spatial cues
• Test2: 5% (10,000) — high-level semantic/cognitive instructions only

The core evaluation protocol computes the success rate for samples where the predicted net camera movement falls within a fixed tolerance (e.g., $t=1,\ldots,k$3) in both pitch and yaw:

$t=1,\ldots,k$4

Model benchmarking in the SaPaVe work demonstrates ~16% average improvement in semantic camera control over baseline MLLMs such as Qwen2.5-VL-72B, Multi-SpatialMLLM, and Gemini-2.5-Pro (Liu et al., 12 Mar 2026).

5. Practical Applications and Model Integration

ActiveViewPose-200K facilitates various core research and application activities:

• Semantic Camera Adapter Training: The dataset enables the construction of semantic camera adapters that can be integrated via LoRA into existing VLMs, providing localized action priors for where to direct attention or focus.
• Pretraining Continuous Next-Best-View Policies: It supports pretraining of policies for camera placements conditioned on language, suitable for robotics, warehouse search, and assistive navigation.
• Generalization and Plug-and-Play Use: With standardized format and objective, the dataset allows drop-in application to evaluate the ability of any vision-LLM to ground arbitrary instructions in egocentric viewpoint transitions.

Potential extensions include generalization to full 6-DoF camera control (translation + rotation), sim-to-real domain adaptation, and integration into mobile manipulation stacks for broad semantic search tasks. This suggests ongoing development toward more complex and realistic embodied manipulation scenarios.

6. Significance, Limitations, and Prospective Directions

ActiveViewPose-200K is distinguished by its curated, instruction-grounded design, supporting learning of semantic, viewpoint-invariant perception with minimal ambiguity. Its combination of realistic scene synthesis, diverse language prompts, and explicit sub-optimal to optimal transitions provides a robust substrate for active perception research.

The restriction to 2-DoF (pitch/yaw) head rotations simplifies the action space but may limit direct applicability to tasks requiring 6-DoF camera control. A plausible implication is that future expansions could incorporate translational degrees of freedom and richer real-world sensory inputs (e.g., RGB-D streams), advancing sim-to-real transfer and mobile manipulation. The semi-automated annotation and manual review pipeline mitigates data quality concerns inherent to large-scale language-driven datasets.

In the experimental context of SaPaVe (Liu et al., 12 Mar 2026), ActiveViewPose-200K demonstrably advances benchmarking and model performance for semantic camera control, establishing robust baselines for comparison and catalyzing further research in active, language-driven perception for robotics and embodied systems.