ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation

Abstract: Recent advances in robot manipulation have leveraged pre-trained vision-LLMs (VLMs) and explored integrating 3D spatial signals into these models for effective action prediction, giving rise to the promising vision-language-action (VLA) paradigm. However, most existing approaches overlook the importance of active perception: they typically rely on static, wrist-mounted cameras that provide an end-effector-centric viewpoint. As a result, these models are unable to adaptively select optimal viewpoints or resolutions during task execution, which significantly limits their performance in long-horizon tasks and fine-grained manipulation scenarios. To address these limitations, we propose ActiveVLA, a novel vision-language-action framework that empowers robots with active perception capabilities for high-precision, fine-grained manipulation. ActiveVLA adopts a coarse-to-fine paradigm, dividing the process into two stages: (1) Critical region localization. ActiveVLA projects 3D inputs onto multi-view 2D projections, identifies critical 3D regions, and supports dynamic spatial awareness. (2) Active perception optimization. Drawing on the localized critical regions, ActiveVLA uses an active view selection strategy to choose optimal viewpoints. These viewpoints aim to maximize amodal relevance and diversity while minimizing occlusions. Additionally, ActiveVLA applies a 3D zoom-in to improve resolution in key areas. Together, these steps enable finer-grained active perception for precise manipulation. Extensive experiments demonstrate that ActiveVLA achieves precise 3D manipulation and outperforms state-of-the-art baselines on three simulation benchmarks. Moreover, ActiveVLA transfers seamlessly to real-world scenarios, enabling robots to learn high-precision tasks in complex environments.

• The paper introduces a coarse-to-fine pipeline that integrates active perception, including adaptive viewpoint selection and 3D zoom-in, to enhance manipulation precision.
• The paper demonstrates that leveraging multi-view 3D rendering and spatial reasoning improves robustness under occlusion and environmental perturbations.
• The paper validates its approach with significant gains on benchmarks such as RLBench and real-robot tasks, achieving up to 41% improvement over baselines.

ActiveVLA: Active Perception in Vision-Language-Action Models for Precise 3D Robotic Manipulation

Motivation and Contributions

ActiveVLA targets the major limitation of passive perception in state-of-the-art Vision-Language-Action (VLA) models for robotic manipulation. Existing VLA models, while leveraging powerful vision-language backbone architectures and integrating 3D signals, generally process data from fixed, wrist-mounted cameras. This constraint results in limited adaptability to occlusions and suboptimal information acquisition, particularly for long-horizon and fine-grained tasks. The paper proposes a framework that injects active perception—specifically adaptive viewpoint selection and 3D zoom-in—into VLA models, enabling robots to autonomously optimize their sensory input for more robust and precise manipulation.

The framework is built as a coarse-to-fine pipeline, underpinned by multi-view 3D rendering and spatial reasoning modules. The authors demonstrate strong empirical gains over leading baselines in simulation and real-robot settings, underscoring the central claim: active perception mechanisms, tightly coupled with state-of-the-art VLM backbones, produce substantial improvements in manipulation performance, generalization, and robustness under occlusion-heavy and perturbed environments.

Coarse-to-Fine Active Perception Pipeline

ActiveVLA’s pipeline begins with reconstructing a 3D point cloud of the workspace from calibrated RGB-D observations. The model follows a two-stage approach:

• Coarse stage: Three orthographic projections (top, front, right) are rendered and processed together with a language instruction by the PaliGemma VLM backbone. Heatmap localization predicts task-relevant 2D regions, which are back-projected to 3D to locate the most salient workspace subvolumes (Figure 1).

  Figure 1: ActiveVLA pipeline integrates multi-view rendering and 3D heatmap localization before active viewpoint selection and 3D zoom-in.

• Fine stage: The agent initiates active view selection by generating candidate camera poses around the identified region, optimized for visibility, distance, and geometric diversity. The most informative view is selected, followed by an active 3D zoom-in—effectively narrowing the field of view to maximize resolution on the focus region, simulating an optical zoom in virtual rendering space. Successive refined heatmap predictions from the zoomed-in view enable high-precision keypoint localization and 6D pose estimation.

  Figure 2: Qualitative manipulation progression: coarse 3D scene projections, heatmap prediction, informed viewpoint planning, and zoom-in for precise interaction.

This hierarchical pipeline strategically balances coverage and resolution, mitigating trade-offs inherent in passive fixed perspectives—enabling dynamic context acquisition and fine-grained detail necessary for manipulation in complex scenes and cluttered environments.

Action Prediction and Policy Structure

Action prediction proceeds from the fused global-local representations encoded by the VLM. Multi-view 2D heatmaps are back-projected into the 3D feature grid, yielding a score volume over discretized workspace points. Translational action targets are selected as maxima in this score volume. Orientation is discretized (Euler angles) and inferred via hierarchical fusion across global semantic and local ROI features.

The policy predicts a keyframe action tuple comprising 6-DoF pose, gripper state, and binary collision flag. By tightly coupling global semantic context (via max-pooled encoder features) and local fine-grained appearance (via ROI-aware sampling), the architecture supports robust, multitask manipulation with high spatial precision—even under severe occlusion and non-i.i.d. viewpoint distribution.

Experimental Validation and Numerical Results

Evaluation is performed on RLBench (18 tasks), COLOSSEUM (robustness under 12 perturbation categories), and GemBench (compositional skill generalization), as well as real-robot tasks featuring intrinsic occlusion, clutter, and fine-grained reasoning (Figures 3, 5, 6, 7).

Figure 3: ActiveVLA actively perceives and manipulates in occlusion-heavy, spatially complex tasks.

Key results:

• RLBench: 91.8% average success rate, first place in 10 tasks, outperforming all baselines with substantial margin in precision- and contact-critical scenarios.
• COLOSSEUM: 65.9% mean success rate, 1.9% absolute gain over previous best, robust under color, texture, size, lighting, and viewpoint perturbations.
• GemBench: 51.3% average, state-of-the-art performance across core action primitive levels.
• Real-robot tasks: 92–95% success rate across heavily occluded tasks (e.g., towel retrieval, stacking, cluttered grasping, fine-grained extraction). Quantitative gains over TriVLA and RVT-2 reach up to 41%.

  Figure 4: Ablation curves: success rates rise with increased active views and zoom-in factors, up to a trade-off with computational cost.

These empirical findings buttress the core claims: active perception modules (view selection and zoom-in) consistently yield success rate gains across all evaluated domains, with strong robustness to structured environmental changes and perturbations.

Implications, Limitations, and Future Directions

The introduction of active perception into VLA architectures marks a decisive step toward context-adaptive embodied intelligence. Practical implications include:

• Enhanced generalization: Dynamic viewpoint planning and local detail acquisition support transfer to unseen occluded/perturbed scenarios without retraining.
• Framework extensibility: The approach can be integrated with multi-agent coordination, tactile-visual fusion, and reinforcement-based online adaptation.
• Real-robot reliability: Improved robustness under non-i.i.d. sensor noise, environmental dynamics, and real-world occlusion structures.

Theoretical ramifications extend to bridging 3D spatial reasoning with language-conditioned policy learning, and advancing closed-loop perception-action cycles in embodied agents. Open challenges remain in scaling active perception pipelines to multi-sensor, multi-actuator systems, optimizing trade-offs in computation vs. perception, and incorporating interaction history for long-horizon planning.

Figure 5: 18 RLBench tasks: broad distribution of manipulation challenges used for evaluation.

Figure 6: Direct manipulation tasks with varying physical interaction modalities.

Figure 7: Examples of occlusion-rich, spatially complex real-world tasks, highlighting the necessity of active viewpoint and zoom planning.

Conclusion

ActiveVLA establishes a new paradigm by actively integrating perception in vision-language-action robotic frameworks. Through coarse-to-fine spatial reasoning, multi-view selection, and zoom-in modules coupled with powerful VLM backbones, ActiveVLA attains state-of-the-art performance, strong robustness, and reliable transfer for precision manipulation tasks. Future directions include multisensory extension, scalable hierarchical skill composition, domain adaptation, and joint policy optimization under closed-loop perception-action. ActiveVLA’s adaptive observation strategy constitutes a foundational step for embodied agents toward universal manipulation in occluded, dynamic, and unstructured environments (2601.08325).