---
title: 'NeU-NBV: Uncertainty-Driven NBV for Robotics'
url: https://www.emergentmind.com/topics/neu-nbv-framework
type: topic
---

# NeU-NBV: Uncertainty-Driven NBV for Robotics

The NeU-NBV Framework is a paradigm for active perception in robotics that addresses the problem of next-best-view (NBV) planning for scene exploration and self-localization under domain shift. It combines a mapless information-seeking approach grounded in uncertainty-aware neural rendering ("NeU-NBV" in [2303.01284]) and a domain-invariant, cue-driven RL policy for cross-domain self-localization ("Domain-invariant NBV Planner for Active Cross-domain Self-localization" [2102.11530]). The core innovation is to drive the view acquisition policy by maximizing information value—either in terms of predicted renderer uncertainty or robust, domain-invariant landmarks—rather than heuristic or explicit 3D map construction.

## 1. System Architecture and NBV Problem Formulation

The NeU-NBV framework formalizes NBV selection as an iterative, data-driven process in which the system maintains:

- A growing reference set $\mathcal{I}$ of RGB images with associated camera poses.
- An image-based neural renderer $f_\theta$, trained offline on diverse scenes and fixed at test time.

At each acquisition step, a discrete candidate set of views $\mathcal{V} = \{v_k \mid k=1,\ldots,K\}$ is sampled within neighborhood constraints (bounded azimuth/elevation). For each $v_k$, the $N$ closest existing references in pose space are selected, and the renderer predicts an uncertainty map $U_k$ for all pixels and channels. The mean uncertainty $g(v_k)$ is computed:

$$
g(v_k) = \frac{1}{H_r W_r 3} \sum_{x \in \text{pixels}} U_k(x)
$$

The NBV is selected by maximizing this criterion: $v^* = \arg\max_{v_k \in \mathcal{V}} g(v_k)$.

No explicit 3D map is constructed or updated. Instead, the approach uses the internal uncertainty of a photometric renderer as a proxy for unexplored or ambiguous regions of the scene. After capturing the real image at $v^*$, the observation is added to $\mathcal{I}$, and the process continues until the measurement budget $B$ is exhausted.

A related, domain-invariant variant [2102.11530] targets active self-localization under changing appearance (season, weather). Here, the architecture includes:

- A multi-scale pole-like landmark detector (PLD) CNN, yielding a compact 4-dimensional feature $f_t$ summarizing the likelihood of domain-stable geometric cues.
- A lightweight deep Q-network (DQN) policy $\pi: f_t \mapsto a_t$ trained to maximize pole detection rates while minimizing movement cost.
- An experience replay buffer and Bag-of-Words pose retrieval module.

The pose-estimation process is triggered opportunistically when sufficient landmark cues are detected.

## 2. Neural Rendering and Uncertainty Estimation

NeU-NBV builds on PixelNeRF but incorporates two critical changes:

1. **Adaptive ray-marching LSTM:** Instead of dense volumetric sampling, an LSTM dynamically determines the next sample point along each ray, leveraging previous feature aggregation for efficient view synthesis.
2. **Probabilistic output head:** For each pixel, the network predicts both the logit-space mean $\mu_i$ and standard deviation $\sigma_i$ for each color channel. The RGB channel $c_i$ is modeled as logistic-normal:

$$
z_i = \mathrm{logit}(c_i) \sim \mathcal{N}(\mu_i, \sigma_i^2)
$$

This enables direct aleatoric per-pixel uncertainty estimation without ensembles or dropout. At inference, per-pixel uncertainty $u_i$ is computed as the variance of sigmoid-transformed samples drawn from $\mathcal{N}(\mu_i, \sigma_i^2)$.

The network is trained using the negative log-likelihood of the logistic-normal model:

$$
\mathcal{L}_\textrm{photo} = \sum_{i=1}^3 \left[ \frac{1}{2} \ln(\sigma_i^2) + \ln(y_i(1-y_i)) + \frac{(\operatorname{logit}(y_i)-\mu_i)^2}{2 \sigma_i^2} \right]
$$

No additional regularization, depth supervision, or adversarial loss is applied.

## 3. NBV Selection Algorithm and Planning Loop

At runtime, NeU-NBV executes the following procedure until the acquisition budget $B$ is reached:

```
Input: pretrained renderer f_theta, initial reference set I, budget B, candidate count K, nearest refs N
for step = 1 to (B - |I|):
    V = sample_K_candidate_views(current_pose, K)
    best_score = -∞
    best_view  = None

    for v in V:
        refs = find_N_closest_refs(I, v, N)
        U = f_theta.predict_uncertainty(v, refs)
        score = mean(U)   # average over H_r x W_r x 3
        if score > best_score:
            best_score = score
            best_view  = v

    (I_new, T_new) = capture_image(best_view)
    I.append((I_new, T_new))
end
return I
```

This policy requires only local operations (nearest neighbor pose search, feedforward inference, empirical averaging) and is mapless—no volumetric or geometric scene model is built or maintained. Because $f_\theta$ is pretrained, there is no per-scene retraining.

## 4. Domain-Invariant NBV for Active Self-Localization

The variant in [2102.11530] introduces several components to address visual domain shifts:

- **Pole-like Landmark Detector (PLD):** A multi-encoder CNN inspired by HED, trained on pole endpoint annotations, robustly detects pole-like structures that are invariant to appearance variations.
- **Spatial Landmark Aggregation (SLA):** The PLD's output is binned horizontally and aggregated to form $f_t \in \mathbb{R}^4$.
- **Deep Q-Learning Policy:** A model-free DQN maps $f_t$ to discrete forward motion actions $A$. Rewards favor observations where pole cues are detected and penalize unnecessary moves.
- **Passive Self-Localization (PSL):** Upon pole detection, a Bag-of-Words-based retrieval estimates pose by matching $f_t$ to a database.
- **Domain Generalization:** The PLD is pretrained on a source domain and transfers directly without domain adaptation or adversarial losses. Mapping policy evaluation to a compact geometry-driven feature enables robust performance across environmental changes.

## 5. Experimental Protocols and Benchmark Results

**Datasets:**

- Real: DTU multiview stereo (49 views/scene; 88 train, 15 test), 400x300 px.
- Synthetic: ShapeNet (car, moto, camera, ship), 100 views/object, 200x200 px.
- Domain-invariant NBV: University of Michigan NCLT dataset, four seasons, 26k images/sequence.

**Training:**

- NeU-NBV: Adam, LR $1 \times 10^{-5}$; LSTM sampling iterations $T=16$; 2 days on one RTX A5000; 3-5 random reference views/scene.
- DQN NBV: $\gamma=0.99$, Adam, batch size 32, buffer size $10^5$, target update every 1k steps; exploration temperature annealed from 1.0 to 0.1.

**Evaluation:**

- **Uncertainty Calibration:** Spearman's Rank Correlation (SRCC) between predicted uncertainty and true MSE; Area Under Sparsification Error (AUSE).
    - Aleatoric uncertainty SRCC: $\approx 0.84$ (competing methods: $0.27$–$0.83$); AUSE: $\approx 0.12$ (vs.\ $0.26$–$0.50$).
- **Planning Quality:** Test-time PSNR and SSIM on held-out images after fixed-budget planning (DTU: 9 images, ShapeNet/indoor: 20 images, 50 candidates/step).
    - Uncertainty-based NBV outperforms random and max-distance planners on both DTU and simulator setups.
- **Impact on Downstream Reconstruction:** Instant-NGP trained on data acquired by NeU-NBV yields higher PSNR/SSIM than models trained on random or max-distance acquisitions.
- **Domain Transfer in NBV DQN:** Median rank of ground-truth pose $\approx1.2$ after $\approx5$ moves with learned policy; baseline heuristics yield rank $\approx2.8$ after $\approx6$ moves.

## 6. Strengths, Limitations, and Future Prospects

**Strengths:**
- NeU-NBV achieves efficient, mapless, uncertainty-driven view planning with no per-scene retraining or explicit 3D map construction.
- Uncertainty estimates are strongly correlated with actual reconstruction error, enabling effective budget utilization.
- Domain-invariant variant leverages geometry-driven cues, providing robust NBV policies across seasons/lighting without retraining.

**Limitations:**
- In domain-invariant NBV, the PLD may confound vertical structures unrelated to poles, especially in cluttered scenes.
- Neither approach explicitly handles full occlusions, e.g., poles obstructed by dynamic obstacles.
- The reward structure in RL-based variant is sparse; more informative shaping (e.g., retrieval-score gain) could accelerate learning.
- The view planner's action space is restricted (e.g., forward motion only) in the RL variant.

**Future Directions:**
- Integrating richer or multi-cue representations (e.g., combining geometric and photometric uncertainty) could further extend robustness.
- Mechanisms for active disambiguation under occlusion or to support higher-dimensional navigation policies are natural extensions.
- Exploration of adversarial or contrastive domain-alignment methods may further improve invariance.

## 7. Context within Active Perception and Neural Rendering

The NeU-NBV framework represents a synthesis of active perception, deep photometric rendering, and robust landmark-based reasoning. By eschewing explicit geometric models in favor of information-driven rendering uncertainty or domain-invariant geometric cues, the framework addresses key bottlenecks of earlier NBV planners: computational scalability, sensitivity to domain shift, and sample efficiency. It contributes both a practical methodology for data acquisition in scene understanding and a benchmark for uncertainty-driven planning in neural rendering and robotic self-localization tasks.

Source: https://www.emergentmind.com/topics/neu-nbv-framework