Closed-Loop NBV Planning

• Closed-loop NBV planning is an iterative sensor control approach that conditions each new viewpoint on all prior measurements to maximize task-driven information gain.
• It integrates real-time sensing, model updates, candidate view generation, and utility optimization to enable efficient and precise 3D reconstructions.
• Feedback cycles allow the system to balance motion costs and predicted gains, improving coverage and performance in both single-agent and multi-agent setups.

Closed-loop next-best-view (NBV) planning is a class of iterative sensor control strategies where the selection of each subsequent sensor viewpoint is conditioned on all measurements accumulated to date. The core objective is to maximize task-driven information gain (e.g., coverage, object recognition certainty, affordance for manipulation) while minimizing costs such as motion, time, or redundancy. This approach underpins active 3D reconstruction, autonomous exploration, and intelligent manipulation in robotics and computer vision.

1. Formal Problem Statement and Optimization Objectives

Closed-loop NBV planning is defined over a sequence of sensor states indexed by time or iteration. The system (e.g., a robot or team of robots) incrementally acquires data $S_t$ corresponding to the observed portion of a scene, object, or environment. The NBV problem is then, at step $t$, to select the next viewpoint(s) $X^* \subset \Phi$ maximizing expected information gain while accounting for control costs and task constraints.

Consider a multi-robot scenario with $n$ 3D sensing agents $r_1, \ldots, r_n$:

• Let $\Phi$ be the discrete set of admissible candidate viewpoints.
• The current observed set is $S_t$—the union of all data acquired thus far.
• For any robot whose current pose is $\phi^t_{r_j}$, the control cost to a candidate view $\phi_{r_j}$ is

$C(X) = \sum_{j=1}^n d(\phi^t_{r_j}, \phi_{r_j})$

• The (expected) information gain is typically expressed as

$$G(X) = \Bigl|\bigcup_{j=1}^n \left(\operatorname{VisPred}(\phi_{r_j}) \setminus S_t \right)\Bigr|$$

where $\operatorname{VisPred}(\phi_{r_j})$ is the set of predicted surface points visible from $\phi_{r_j}$ as inferred from a completion model.

Classical approaches scalarize the trade-off via a utility function $U(X) = \alpha G(X) - \beta C(X)$ and optimize

$X^* = \arg\max_X U(X)$

In the MAP-NBV framework (Dhami et al., 2023), a constrained problem is solved:

$$\min_{X \subset \Phi} C(X) \quad \text{s.t.} \quad G(X) \geq \tau G_{\max}$$

where $G_{\max}$ is the maximal information gain among candidates and $\tau\in(0,1]$ enforces a minimal progress threshold.

Variants and adaptations of this framework exist for single-agent, multi-agent, target-driven grasping, or classification tasks. The cost and utility functions may include model-specific terms (e.g., expected classification margin (Korbach et al., 2021), grasp quality (Zhang et al., 2023, Breyer et al., 2022), or entropy reduction (Lodel et al., 2022)).

2. Core Algorithmic Structures: Feedback and Iteration

All closed-loop NBV planners establish tight measurement-planning-actuation cycles:

1. Sensing: Acquire new raw sensory data (e.g., depth, RGB-D, LiDAR).
2. Map/model update: Integrate the new data into a current scene representation (TSDF, voxel grid, point cloud, or implicit neural field).
3. Candidate generation: Sample a set of candidate viewpoints as feasible next NBV options, considering robot kinematics, workspace, and object/environment geometry.
4. Prediction/estimation: Use explicit geometric reasoning or learned priors to estimate visibility, information gain, or task-relevant uncertainty for each candidate.
5. Objective evaluation and selection: Score candidates with respect to a principal utility function balancing gain, task success, and cost; select the next NBV.
6. Execution and feedback: Plan and execute the motion; repeat the loop using the now-augmented observation set.

Termination is commonly triggered by coverage convergence, task success (e.g., stable grasp), or marginal information gain falling below a threshold.

3. Representation of Scene and Prediction of Unobserved Geometry

A critical component of modern NBV frameworks is the internal representation of observed and unobserved geometry.

• Occupancy Grids / TSDFs: Volumetric models (e.g., OctoMap, TSDF grids) are updated at each step. Candidate viewpoints’ expected gains are computed by ray-casting or visibility analysis. Used in (Dhami et al., 2023, Breyer et al., 2022, Zhang et al., 2023, Jia et al., 18 Jan 2025).
• Point Clouds and Completion Networks: Partial clouds are extended via learned completion networks (e.g., PoinTr-C transformer, (Dhami et al., 2023, Dhami et al., 2023)). Completed models provide rich priors on hidden structure for predictive view planning.
• Implicit Neural Fields: Continuous occupancy networks parameterized by neural networks enable uncertainty and entropy evaluation via differentiable rendering (Yan et al., 2023).
• Ellipsoids/Frontiers: For computational efficiency, voxel clusters may be compressed into ellipsoidal representations for projection-based visibility scoring (Jia et al., 18 Jan 2025, Jia et al., 18 Sep 2024).

In all cases, visibility or gain from a candidate NBV is estimated by mapping the predicted or modeled scene from the candidate's perspective, utilizing geometric or statistical models as appropriate.

4. Viewpoint Candidate Scoring: Information Gain, Prediction, and Cost

The principal criterion for NBV selection is a utility function quantifying anticipated information gain against cost. Common strategies include:

• Prediction-guided gain: Quantify the number of new surface points expected to be visible, derived from explicit completion (Dhami et al., 2023, Dhami et al., 2023), implicit uncertainty (Yan et al., 2023), or “frontier” voxels (Jia et al., 18 Jan 2025).
• Projection-based scoring: Compress candidate observation regions to ellipsoid clusters; score candidates by their projected pixel coverage on the image plane, weighted by depth ordering for occlusion handling (Jia et al., 18 Jan 2025, Jia et al., 18 Sep 2024).
• Entropy and uncertainty metrics: Maximize predicted scene entropy/decrease in occupancy grid uncertainty (Yan et al., 2023, Lodel et al., 2022).
• Task-specific criteria: Grasp quality scoring (Zhang et al., 2023, Breyer et al., 2022), classifier confidence margins (Korbach et al., 2021), or fruit ROI coverage rates (Zaenker et al., 2023).

Cost terms typically include travel/path length, control effort, or motion time, and may be explicitly penalized or incorporated via constraints.

5. Coordination and Multi-Agent Mechanisms

In multi-robot settings, closed-loop NBV planning benefits from centralized or decentralized coordination:

• Centralized fusion and joint assignment: Robots synchronize local map updates and exchange partial clouds, fusing observations to maintain a global model (Dhami et al., 2023).
• Joint utility optimization: Rather than independent NBV choices, the union of predicted NBV gains is optimized to minimize view redundancy and maximize coverage per time or distance (Dhami et al., 2023).
• Assignment strategies: For small teams, exhaustive assignment may be computationally tractable. For larger teams, sequential greedy assignment subject to global gain constraints (e.g., select robot 1's NBV to maximize marginal team gain, then robot 2, etc.) is typical (Dhami et al., 2023).

Decentralized or distributed representations and assignment schemes are less explored in the existing literature, but findings highlight significant coverage improvements from joint information-theoretic planning versus myopic or greedy selection.

6. Computational Efficiency and Practical Advantages

Efficiency is a central concern in NBV planning, as explicit ray-casting is often the computational bottleneck. Recent works have introduced significant algorithmic accelerations:

• Projection-based evaluation: By fitting and projecting a small number of ellipsoids (rather than evaluating all voxels or surfels), the computational cost per candidate reduces dramatically. PB-NBV achieves $5.10\,\mathrm{s}$ per iteration with $98.4\,\%$ coverage, whereas traditional volumetric and ray-casting methods require $14$–$27\,\mathrm{s}$ per iteration for $73$–$89\,\%$ coverage (Jia et al., 18 Jan 2025).
• Continuous optimization: Differentiable view utility functions (e.g., in implicit neural fields) permit direct gradient ascent in pose space, bypassing enumeration of candidate sets and enabling fine-grained local optimization (Yan et al., 2023).
• Frontier graph pruning and partition strategies: Partitioning the candidate hemisphere and enforcing sequential region coverage prevent backtracking and ensure robust model registration (Jia et al., 18 Jan 2025, Jia et al., 18 Sep 2024).
• Shadowcasting and “fast gain” computation: Recursive shadowcasting algorithms and cuboid-based gain evaluations achieve $6$–$20\times$ faster per-step computation versus standard ray-tracing, supporting real-time NBV for exploration (Batinovic et al., 2021).

These enhancements make closed-loop NBV feasible for deployment on resource-constrained platforms and allow rapid adaptation in dynamic or partially known environments.

7. Empirical Results, Limitations, and Extensions

Empirical evaluations consistently demonstrate that closed-loop NBV planners—especially those using predictive models and joint multi-agent selection—achieve higher coverage, faster convergence, and/or improved task completion rates compared to non-predictive, heuristic, or single-step approaches. Specific metrics include:

• $19\%$ improvement in reconstructed points over classical multi-agent planners in AirSim and ShapeNet (Dhami et al., 2023)
• $22.75\%$ more new points in first iteration versus single-agent prediction-only NBV (Dhami et al., 2023)
• $98.4\%$ point cloud coverage at $5.10$ s/iteration for PB-NBV (Jia et al., 18 Jan 2025)
• In real-world object scanning, PB-NBV achieves $>90\%$ coverage in 8 steps with stable per-step timing across objects (Jia et al., 18 Jan 2025)

Limitations are generally domain-specific and include: dependence on segmentation quality and object stationarity, potential failure of visibility estimation in highly concave or self-occluding geometries, challenge of model misalignment without robust registration, increased planning times for high-resolution models, and the requirement for full cooperation in multi-agent cases.

Potential future directions include online uncertainty modeling of predictive completions, continuous action-space NBV via reinforcement learning, adaptation to dynamic scenes, learned collision predictors, and autonomous object discovery without bounding box priors.

---

In summary, closed-loop NBV planning defines a feedback-driven, iterative trajectory selection approach for active perception. By leveraging up-to-date observations, state-of-the-art predictions of unobserved geometry, and multi-objective optimization over information gain and action cost, these systems outperform classical NBV strategies in coverage, convergence, and efficiency, establishing a rigorous paradigm for autonomous scene exploration and reconstruction (Dhami et al., 2023, Jia et al., 18 Jan 2025, Yan et al., 2023).