View Planning in Robotics

• View Planning (VP) is the process of selecting optimal sensor poses to observe a scene or object for high-quality reconstruction and task performance.
• VP methods employ set-cover optimization, reinforcement learning, and one-shot prediction to balance coverage, motion cost, and reconstruction fidelity.
• Recent research integrates view planning with real-time constraints and downstream tasks, emphasizing active sensing in robotic and imaging applications.

View planning (VP) denotes the problem of choosing a sequence or set of sensor poses from which a robot or camera system should observe a scene or object so that geometry, appearance, or task-relevant state can be recovered as well as possible under costs and constraints such as view budget, motion budget, and reachability. In object-centric active 3D reconstruction, VP is typically posed on or around a sphere centered at the object; in model-based inspection it is often formalized as a set covering optimization problem over visible surface elements; and in recent visual-planning work the term also appears in image-space or vision-conditioned formulations that plan directly from views rather than over explicit 3D maps (Pan et al., 11 May 2026, Kaba et al., 2016, Liu et al., 2020).

1. Formal problem classes

A classical formulation is the model-based View Planning Problem (VPP): given a known 3D mesh model $\Omega$ and a finite set of camera poses $\mathbb{S}=\{(\ell_i,d_i)\}$, find a minimum-cardinality subset whose visible submeshes sufficiently cover the object. In set-cover notation, if $S$ is the discretized surface and each viewpoint induces a subset $S_i\subseteq S$, the objective is

$$\min_{J\subseteq I}|J| \quad\text{subject to}\quad S=\bigcup_{j\in J}S_j.$$

This framing makes VPP a special case of set covering optimization, hence NP-hard, and explains the historical use of greedy coverage baselines and coverage thresholds such as the relative coverage criterion (RCC) (Kaba et al., 2016).

A second class arises in active 3D reconstruction, where the object is not assumed known at test time. Here VP is a next-best-view or view-budgeted acquisition problem coupled to a reconstruction model. PVP-Recon, for example, formulates the problem as active image-based SDF reconstruction under a fixed budget, starting from as few as 3 RGB views with known camera poses and intrinsics, then selecting additional poses to maximize expected reconstruction improvement (Ye et al., 2024). Active-HOF similarly plans views over a hemispherical viewing space $\mathcal{W}$ for a single externally visible object, but uses a learned Multi-HOF reconstruction as the proxy geometry on which visibility is computed (Engin et al., 2019).

A third class is deployment-oriented object-centric VP, where evaluation itself is part of the formalism. ObjView-Bench separates three quantities that earlier studies often conflated: omnidirectional self-occlusion as an object-side visibility ceiling, observation saturation difficulty as the number of omnidirectional views needed before marginal gains become negligible, and protocol-dependent planning difficulty as the minimum set-cover number induced by a fixed candidate-view protocol (Pan et al., 11 May 2026). This separation is important because poor performance may reflect an object’s self-occlusion or reachable-view restrictions rather than a planner’s selection strategy.

2. Utility functions, visibility models, and optimization signals

VP methods differ most sharply in the signal they optimize. In model-based coverage planning, the canonical signal is additional covered surface area. A notable refinement is the score

$$f_\lambda(X)=\frac{\mathcal{A}(X)}{\mathcal{L}(X)^\lambda},$$

where $\mathcal{A}(X)$ is covered area and $\mathcal{L}(X)$ is boundary length. When $\lambda=0$, this reduces to greedy area maximization; when $\mathbb{S}=\{(\ell_i,d_i)\}$0, it penalizes long, ragged frontiers and therefore discourages leaving small uncovered islands that later require many cleanup views (Kaba et al., 2016).

In reconstruction-coupled VP, visibility alone is often treated as insufficient. Active-HOF reports that visible parts of the predicted object have lower occupancy entropy than the full voxel grid, and therefore uses visibility over the current reconstructed surface as a proxy for reconstruction quality (Engin et al., 2019). PVP-Recon goes further by defining a warping-consistency score

$\mathbb{S}=\{(\ell_i,d_i)\}$1

where a rendered candidate image is depth-warped into its nearest existing training view, and the score measures masked photometric inconsistency. In practice, high scores often correspond to views that expose previously unseen regions, so the score acts as a proxy for information gain without physically capturing the image first (Ye et al., 2024).

Agricultural VP work replaces coverage proxies with direct reconstruction-quality objectives. One line defines a stereo-geometric reward based on triangulation baselines and field-of-view feasibility, then treats the noisy objective as a black-box function optimized by Bayesian optimization under unknown environmental perturbations (Bacharis et al., 2023). BOSfM makes the reconstruction loop even more explicit by defining

$\mathbb{S}=\{(\ell_i,d_i)\}$2

where $\mathbb{S}=\{(\ell_i,d_i)\}$3 is the SfM reconstruction produced by a candidate multi-camera configuration $\mathbb{S}=\{(\ell_i,d_i)\}$4, and $\mathbb{S}=\{(\ell_i,d_i)\}$5 is the Chamfer distance to a reference point cloud (Bacharis et al., 28 Sep 2025).

Static LiDAR VPP introduces yet another utility family. VF-Plan computes a Visibility Field whose scalar value at a location is the total valid observed angle over all visible wall segments, after enforcing range bounds and occlusion checks. It then couples coverage with overlap-based registrability and graph compactness via edge weights $\mathbb{S}=\{(\ell_i,d_i)\}$6 and the Weighted Average Path Length (WAPL), producing viewpoint networks that are not only covering but also strongly connected for registration (Xionga et al., 3 Mar 2025).

Paradigm	Core signal	Primary emphasis
Set-cover VPP	Additional coverage or $\mathbb{S}=\{(\ell_i,d_i)\}$7	Minimum views with compact frontier
SDF NBV	Warping score $\mathbb{S}=\{(\ell_i,d_i)\}$8	Information gain for sparse-view reconstruction
SfM-driven BO	Negative Chamfer reward	Direct reconstruction quality
Appearance-based RL	BoV vocabulary change	Visual novelty without geometry in loop
Static LiDAR network design	Overlap graph and WAPL	Registrable, connected scan networks

These objective functions encode a persistent divide in VP research. Some methods optimize observability directly; others optimize a surrogate for future reconstruction fidelity; still others optimize downstream task performance, as in manipulation-oriented active view selection. This suggests that “best view” is task-relative rather than absolute.

3. Major algorithmic paradigms

Greedy coverage remains the foundational algorithmic template because of the set-cover structure. The reinforcement-learning approach of (Kaba et al., 2016) does not discard greedy search; instead, it learns when to use different values of $\mathbb{S}=\{(\ell_i,d_i)\}$9 in the score $S$0. The action space is a small discrete set of $S$1 values, the state is the binary vector of already selected cameras, the reward is a constant per-view penalty $S$2, and SARSA, Watkins-Q, and TD with function approximation are used to reduce the number of required views relative to purely greedy selection.

Appearance-based VP removes geometry from the planning loop. Bag-of-Views (BoV) represents each captured view by SIFT descriptors assigned to regional vocabularies indexed by viewpoint sectors, and defines reward through the change in those vocabularies. A Soft Actor-Critic policy then chooses the next UAV pose to maximize cumulative BoV-change rewards, treating reconstruction quality as a downstream evaluation rather than an online planning variable (Gazani et al., 2023). This approach is explicitly motivated by cases where no CAD model or partial reconstruction is available or affordable during flight.

Graph-based image-space planning forms another lineage. Hallucinative Topological Memory keeps SPTM’s idea that image observations are graph nodes and learned short-horizon reachability defines graph edges, but replaces the original binary connectivity classifier with a contrastive predictive coding energy model and uses a conditional VAE to hallucinate nodes for unseen contexts, enabling zero-shot planning in new domains (Liu et al., 2020). Planning then reduces to shortest-path search in a graph whose nodes remain directly interpretable images.

Recent multimodal work has begun to study view planning itself as a visual reasoning capability. ViewSuite formalizes interactive view planning as 6-DoF camera control in ScanNet scenes and shows that frontier VLMs possess basic single-step view-action knowledge but fail to compose it into multi-turn plans. Its self-exploration plus view-graph distillation framework improves Qwen2.5-VL-7B from 2.5% to 47.8% on interactive view planning, surpassing GPT-5.4 Pro at 18.5% and Gemini 3.1 Pro at 21.4% (Wang et al., 28 May 2026). ING-VP, a separate interactive game benchmark for image-based planning, reports that the best-performing model, Claude-3.5 Sonnet, reaches only 3.37% average accuracy across its settings, indicating that current MLLMs remain weak even on short-horizon spatial planning from views (Zhang et al., 2024).

One-shot set prediction is a distinct paradigm. OSVP predicts an entire subset of views in one forward pass, using a refined dense point cloud from POCO and a binary view-state vector as input. Because the labels are generated by set-cover optimization over dense points refined by an implicit model, the planner is trained to choose a small subset of views sufficient for implicit reconstruction rather than to maximize explicit observed coverage (Hu et al., 2023).

4. Reconstruction-coupled VP systems

Active-HOF exemplifies VP with learned global shape priors. Multi-HOF maps one or more masked RGB images to an implicit point-based object reconstruction, and the planner greedily chooses the next viewpoint that maximizes coverage of previously unseen predicted surface voxels. The dynamic version, which updates the reconstruction after each new image, requires fewer views than a static first-view-only reconstruction, especially at high coverage thresholds, and substantially outperforms an information-maximization baseline built on occupancy uncertainty (Engin et al., 2019).

PVP-Recon tightly interleaves planning and neural implicit reconstruction. It initializes from 3 clustered views, trains the current SDF and color fields for 1000 iterations, evaluates all remaining candidate poses by the warping score, selects the highest-scoring pose, adds the real image, and repeats until a strict budget is reached. On DTU, “planning” yields mean Chamfer distance 0.81 mm, compared with 0.90 for cluster views, 0.97 for random, and 1.01 for farthest sampling; with 6 views on DTU, the paper reports Entropy at 0.93/59 s, NeurAR at 1.23/6.5 s, NeU-NBV at 1.12/11.1 s, and PVP-Recon at 0.81/7.8 s (Ye et al., 2024). The reconstruction module’s progressive hash activation and directional Hessian loss are not secondary details: they stabilize early geometry so that the warping-based NBV signal is not dominated by overfitting artifacts.

OSVP shows the same reconstruction-coupled logic in a one-shot form. It uses POCO to refine an initial sparse observation into a dense point cloud, then predicts a binary mask over a 32-view candidate sphere. On the reported test objects, the proposed method achieves $S$3 surface coverage with $S$4 views and $S$5 m movement cost, while delivering substantially higher coverage per view and coverage per meter than search-based NBV and earlier one-shot baselines (Hu et al., 2023). The central claim is not that explicit coverage is maximized, but that implicit reconstruction quality can remain high even when many low-gain cleanup views are omitted.

Agricultural VP makes the coupling between planning and final geometry explicit at the objective level. The Bayesian-optimization framework of (Bacharis et al., 2023) optimizes a stereo-aware reconstruction score under noisy plant deformations, using an ensemble of Gaussian processes and adaptive Expected Improvement; on 1-, 3-, and 6-plant scenes it reports the lowest Chamfer distance among the compared methods. BOSfM applies the same black-box optimization principle directly to SfM, showing near-zero simple regret in fewer than 50 iterations on its simulated environments and lower Chamfer distance than circle, MCP, and BO-geometric baselines on 3-, 5-, and 6-plant scenes (Bacharis et al., 28 Sep 2025).

Task-aware manipulation extends the same principle beyond reconstruction. TAVP learns a continuous Multi-Viewpoint Exploration Policy that outputs $S$6 camera poses in a look-at spherical parameterization and optimizes them with PPO in a pseudo-environment whose reward is based on downstream grounding and action losses. Combined with a TaskMoE visual encoder, TAVP improves average RLBench success to 86.7% over fixed-view baselines, and its ablation without active exploration collapses to 8.89%, indicating that view selection can be learned as a task-conditioned control variable rather than treated as a static sensor-layout decision (Bai et al., 7 Aug 2025).

5. Evaluation protocols, benchmarks, and deployment constraints

VP evaluation is unusually sensitive to protocol. Sparse-view reconstruction papers typically report datasets such as DTU, BlendedMVS, and Blender, using Chamfer distance, PSNR, and SSIM under strict view budgets; manipulation work often uses RLBench success; LiDAR planning emphasizes viewpoint count, full coverage, overlap, and WAPL; and agricultural work evaluates Chamfer distance and depth-image MAE after SfM or COLMAP (Ye et al., 2024, Bai et al., 7 Aug 2025, Xionga et al., 3 Mar 2025).

ObjView-Bench makes protocol dependence explicit. It contains 5,691 reviewed objects from Objaverse++, with a main pool of 5,523 non-slow-saturation objects and a slow-saturation subset of 168 objects. It evaluates planners under three budget regimes—$S$7, $S$8, and automatic stopping—and under both whole-sphere and quarter-sphere reachable-view settings (Pan et al., 11 May 2026). The benchmark shows that rankings can invert across regimes: at $S$9, iterative NBV-style methods dominate in normalized surface coverage, whereas at $S_i\subseteq S$0, completion-plus-coverage planners and even Random+TSP become competitive because coverage saturates. Reachability also matters materially: on the reported real arm, only about 45.4 out of 128 candidate views are reachable on average, with a range of 26 to 74, and learned planners that output a single discrete view index can fail catastrophically when their chosen view is infeasible (Pan et al., 11 May 2026).

Long-horizon execution introduces another axis absent from many idealized NBV studies. LHVP fixes a quadruped base path and plans an eye-in-hand camera trajectory for a moving manipulator over the full patrol horizon, enforcing per-step arm reachability via a trapezoidal velocity profile and filtering infeasible viewpoints before information-gain evaluation. In both simulation and real experiments it outperforms a “See Nearest” baseline in coverage, and its ablations show that ignoring joint reachability dramatically degrades realized performance (Tankasala et al., 2024). This is an important counterexample to evaluations that assume all candidate viewpoints are equally executable.

Aerial adaptive reconstruction introduces yet another deployment pattern. The adaptive viewing-rectangle method first flies a zigzag exploration path, reconstructs a coarse proxy, then iteratively adds geometry-aligned viewing rectangles whose grids can be covered by a 2.5D TSP tour. In the reported synthetic scenes, a second adaptive pass already improves depth error and completeness substantially over ZigZag, Uniform-Grid, and GVS baselines, and a third pass further improves low-quality regions while adding relatively few views; the paper explicitly reports that three rounds of data collection are sufficient even for very complex scenes (Peng et al., 2018).

6. Conceptual issues, misconceptions, and future directions

A persistent misconception is that VP is synonymous with myopic next-best-view selection. The literature is broader: it includes greedy set cover, reinforcement learning over coverage-compactness trade-offs, one-shot set prediction, image-space graph planning, continuous black-box optimization over camera poses, and long-horizon view-trajectory planning under manipulator dynamics (Kaba et al., 2016, Hu et al., 2023, Liu et al., 2020, Tankasala et al., 2024). This suggests that “VP” is better understood as a family of constrained observation-selection problems than as a single algorithmic template.

A second misconception is that maximizing raw surface coverage is equivalent to maximizing reconstruction quality. Several papers argue otherwise. PVP-Recon optimizes a warping-consistency proxy tied to future SDF quality rather than explicit coverage alone; OSVP shows that implicit priors can recover small missing regions without observing them directly; and BOSfM optimizes the final SfM output through Chamfer distance rather than via geometric visibility heuristics (Ye et al., 2024, Hu et al., 2023, Bacharis et al., 28 Sep 2025). A plausible implication is that coverage remains necessary but is often not a sufficient surrogate once reconstruction, registration, or task execution enters the loop.

A third misconception is that idealized full-sphere evaluations transfer directly to deployment. ObjView-Bench shows that budget regime, reachability constraints, and stopping semantics materially change method rankings and failure modes, while LHVP demonstrates that per-step reachability and continuous motion can dominate realized coverage even when the underlying information objective is unchanged (Pan et al., 11 May 2026, Tankasala et al., 2024).

Current limitations recur across subfields. PVP-Recon reports about 8 seconds per planning round and about 10 minutes total reconstruction in its current implementation, and explicitly notes the need for CUDA-level optimization and extension beyond bounded object-centric scenes (Ye et al., 2024). The RL VPP formulation of (Kaba et al., 2016) trains per object and uses a state encoding tied to specific camera indices, so it does not address cross-object transfer. BoV reports only simulation experiments and notes sensitivity of SIFT-based features to illumination and textureless surfaces (Gazani et al., 2023). VF-Plan assumes a 2D floor plan and static LiDAR. ViewSuite and ING-VP, from a different angle, show that current VLMs still have a substantial multi-turn planning gap in view space (Wang et al., 28 May 2026, Zhang et al., 2024).

Recent terminology also reflects a widening scope. MTR-VP applies “VP” to trajectory planning from surround-view images and routing intent in autonomous driving; unlike classical VP, it does not choose future sensor viewpoints, but it does expose a broader trend toward intent-conditioned planning from views (Keskar et al., 27 Nov 2025). This suggests that the term is now used in two related senses: viewpoint selection for sensing, and planning conditioned on visual observations.

Future directions in the surveyed work are comparatively consistent. They include real-time robotic VP through faster scoring or incremental updates, extension from object-level bounded scenes to large-scale and unbounded environments, integration with mobile robots and drones, reachability-aware learning, difficulty-aware sampling and curricula, hybrid planners that combine learned completion with explicit set-cover optimization, and self-exploration frameworks that distill view graphs into reusable planning priors (Ye et al., 2024, Pan et al., 11 May 2026, Wang et al., 28 May 2026). Across these directions, the central technical trend is clear: VP is moving away from isolated coverage heuristics toward tightly coupled systems in which visibility, reconstruction, embodiment, and task objectives are optimized together.