NeU-NBV: Uncertainty-Driven NBV for Robotics

• The paper introduces a novel framework that selects next-best views by maximizing predicted rendering uncertainty, eliminating the need for explicit 3D map construction.
• It employs an adaptive ray-marching LSTM with a probabilistic output head to efficiently estimate per-pixel uncertainty, enhancing view synthesis and scene reconstruction.
• A domain-invariant variant integrates a pole-like landmark detector and deep Q-learning, enabling robust cross-domain self-localization under varying environmental conditions.

The NeU-NBV Framework is a paradigm for active perception in robotics that addresses the problem of next-best-view (NBV) planning for scene exploration and self-localization under domain shift. It combines a mapless information-seeking approach grounded in uncertainty-aware neural rendering ("NeU-NBV" in (Jin et al., 2023)) and a domain-invariant, cue-driven RL policy for cross-domain self-localization ("Domain-invariant NBV Planner for Active Cross-domain Self-localization" (Tanaka, 2021)). The core innovation is to drive the view acquisition policy by maximizing information value—either in terms of predicted renderer uncertainty or robust, domain-invariant landmarks—rather than heuristic or explicit 3D map construction.

1. System Architecture and NBV Problem Formulation

The NeU-NBV framework formalizes NBV selection as an iterative, data-driven process in which the system maintains:

• A growing reference set $\mathcal{I}$ of RGB images with associated camera poses.
• An image-based neural renderer $f_\theta$, trained offline on diverse scenes and fixed at test time.

At each acquisition step, a discrete candidate set of views $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$ is sampled within neighborhood constraints (bounded azimuth/elevation). For each $v_k$, the $N$ closest existing references in pose space are selected, and the renderer predicts an uncertainty map $U_k$ for all pixels and channels. The mean uncertainty $g(v_k)$ is computed:

$$g(v_k) = \frac{1}{H_r W_r 3} \sum_{x \in \text{pixels}} U_k(x)$$

The NBV is selected by maximizing this criterion: $v^* = \arg\max_{v_k \in \mathcal{V}} g(v_k)$.

No explicit 3D map is constructed or updated. Instead, the approach uses the internal uncertainty of a photometric renderer as a proxy for unexplored or ambiguous regions of the scene. After capturing the real image at $v^*$, the observation is added to $f_\theta$0, and the process continues until the measurement budget $f_\theta$1 is exhausted.

A related, domain-invariant variant (Tanaka, 2021) targets active self-localization under changing appearance (season, weather). Here, the architecture includes:

• A multi-scale pole-like landmark detector (PLD) CNN, yielding a compact 4-dimensional feature $f_\theta$2 summarizing the likelihood of domain-stable geometric cues.
• A lightweight deep Q-network (DQN) policy $f_\theta$3 trained to maximize pole detection rates while minimizing movement cost.
• An experience replay buffer and Bag-of-Words pose retrieval module.

The pose-estimation process is triggered opportunistically when sufficient landmark cues are detected.

2. Neural Rendering and Uncertainty Estimation

NeU-NBV builds on PixelNeRF but incorporates two critical changes:

1. Adaptive ray-marching LSTM: Instead of dense volumetric sampling, an LSTM dynamically determines the next sample point along each ray, leveraging previous feature aggregation for efficient view synthesis.
2. Probabilistic output head: For each pixel, the network predicts both the logit-space mean $f_\theta$4 and standard deviation $f_\theta$5 for each color channel. The RGB channel $f_\theta$6 is modeled as logistic-normal:

$f_\theta$7

This enables direct aleatoric per-pixel uncertainty estimation without ensembles or dropout. At inference, per-pixel uncertainty $f_\theta$8 is computed as the variance of sigmoid-transformed samples drawn from $f_\theta$9.

The network is trained using the negative log-likelihood of the logistic-normal model:

$\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$0

No additional regularization, depth supervision, or adversarial loss is applied.

3. NBV Selection Algorithm and Planning Loop

At runtime, NeU-NBV executes the following procedure until the acquisition budget $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$1 is reached:

$N$1

This policy requires only local operations (nearest neighbor pose search, feedforward inference, empirical averaging) and is mapless—no volumetric or geometric scene model is built or maintained. Because $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$2 is pretrained, there is no per-scene retraining.

4. Domain-Invariant NBV for Active Self-Localization

The variant in (Tanaka, 2021) introduces several components to address visual domain shifts:

• Pole-like Landmark Detector (PLD): A multi-encoder CNN inspired by HED, trained on pole endpoint annotations, robustly detects pole-like structures that are invariant to appearance variations.
• Spatial Landmark Aggregation (SLA): The PLD's output is binned horizontally and aggregated to form $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$3.
• Deep Q-Learning Policy: A model-free DQN maps $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$4 to discrete forward motion actions $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$5. Rewards favor observations where pole cues are detected and penalize unnecessary moves.
• Passive Self-Localization (PSL): Upon pole detection, a Bag-of-Words-based retrieval estimates pose by matching $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$6 to a database.
• Domain Generalization: The PLD is pretrained on a source domain and transfers directly without domain adaptation or adversarial losses. Mapping policy evaluation to a compact geometry-driven feature enables robust performance across environmental changes.

5. Experimental Protocols and Benchmark Results

Datasets:

• Real: DTU multiview stereo (49 views/scene; 88 train, 15 test), 400x300 px.
• Synthetic: ShapeNet (car, moto, camera, ship), 100 views/object, 200x200 px.
• Domain-invariant NBV: University of Michigan NCLT dataset, four seasons, 26k images/sequence.

Training:

• NeU-NBV: Adam, LR $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$7; LSTM sampling iterations $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$8; 2 days on one RTX A5000; 3-5 random reference views/scene.
• DQN NBV: $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$9, Adam, batch size 32, buffer size $v_k$0, target update every 1k steps; exploration temperature annealed from 1.0 to 0.1.

Evaluation:

• Uncertainty Calibration: Spearman's Rank Correlation (SRCC) between predicted uncertainty and true MSE; Area Under Sparsification Error (AUSE).
  • Aleatoric uncertainty SRCC: $v_k$1 (competing methods: $v_k$2–$v_k$3); AUSE: $v_k$4 (vs.\ $v_k$5–$v_k$6).
• Planning Quality: Test-time PSNR and SSIM on held-out images after fixed-budget planning (DTU: 9 images, ShapeNet/indoor: 20 images, 50 candidates/step).
  • Uncertainty-based NBV outperforms random and max-distance planners on both DTU and simulator setups.
• Impact on Downstream Reconstruction: Instant-NGP trained on data acquired by NeU-NBV yields higher PSNR/SSIM than models trained on random or max-distance acquisitions.
• Domain Transfer in NBV DQN: Median rank of ground-truth pose $v_k$7 after $v_k$8 moves with learned policy; baseline heuristics yield rank $v_k$9 after $N$0 moves.

6. Strengths, Limitations, and Future Prospects

Strengths:

• NeU-NBV achieves efficient, mapless, uncertainty-driven view planning with no per-scene retraining or explicit 3D map construction.
• Uncertainty estimates are strongly correlated with actual reconstruction error, enabling effective budget utilization.
• Domain-invariant variant leverages geometry-driven cues, providing robust NBV policies across seasons/lighting without retraining.

Limitations:

• In domain-invariant NBV, the PLD may confound vertical structures unrelated to poles, especially in cluttered scenes.
• Neither approach explicitly handles full occlusions, e.g., poles obstructed by dynamic obstacles.
• The reward structure in RL-based variant is sparse; more informative shaping (e.g., retrieval-score gain) could accelerate learning.
• The view planner's action space is restricted (e.g., forward motion only) in the RL variant.

Future Directions:

• Integrating richer or multi-cue representations (e.g., combining geometric and photometric uncertainty) could further extend robustness.
• Mechanisms for active disambiguation under occlusion or to support higher-dimensional navigation policies are natural extensions.
• Exploration of adversarial or contrastive domain-alignment methods may further improve invariance.

7. Context within Active Perception and Neural Rendering

The NeU-NBV framework represents an overview of active perception, deep photometric rendering, and robust landmark-based reasoning. By eschewing explicit geometric models in favor of information-driven rendering uncertainty or domain-invariant geometric cues, the framework addresses key bottlenecks of earlier NBV planners: computational scalability, sensitivity to domain shift, and sample efficiency. It contributes both a practical methodology for data acquisition in scene understanding and a benchmark for uncertainty-driven planning in neural rendering and robotic self-localization tasks.