OUGS: Active View Selection via Object-aware Uncertainty Estimation in 3DGS (2511.09397v1)

Abstract: Recent advances in 3D Gaussian Splatting (3DGS) have achieved state-of-the-art results for novel view synthesis. However, efficiently capturing high-fidelity reconstructions of specific objects within complex scenes remains a significant challenge. A key limitation of existing active reconstruction methods is their reliance on scene-level uncertainty metrics, which are often biased by irrelevant background clutter and lead to inefficient view selection for object-centric tasks. We present OUGS, a novel framework that addresses this challenge with a more principled, physically-grounded uncertainty formulation for 3DGS. Our core innovation is to derive uncertainty directly from the explicit physical parameters of the 3D Gaussian primitives (e.g., position, scale, rotation). By propagating the covariance of these parameters through the rendering Jacobian, we establish a highly interpretable uncertainty model. This foundation allows us to then seamlessly integrate semantic segmentation masks to produce a targeted, object-aware uncertainty score that effectively disentangles the object from its environment. This allows for a more effective active view selection strategy that prioritizes views critical to improving object fidelity. Experimental evaluations on public datasets demonstrate that our approach significantly improves the efficiency of the 3DGS reconstruction process and achieves higher quality for targeted objects compared to existing state-of-the-art methods, while also serving as a robust uncertainty estimator for the global scene.

• The paper presents a physically-grounded uncertainty model on 3D Gaussian primitives to directly improve view selection for precise object reconstruction.
• It leverages semantic segmentation masks and Jacobian-based covariance propagation to isolate object-level uncertainties and optimize camera views.
• Experimental results show enhanced object fidelity in PSNR, SSIM, and LPIPS, outperforming traditional scene-level uncertainty approaches.

Object-aware Uncertainty Estimation for Active View Selection in 3D Gaussian Splatting

Introduction

The paper "OUGS: Active View Selection via Object-aware Uncertainty Estimation in 3DGS" proposes a new framework for efficiently reconstructing specific objects within complex 3D scenes, focusing on the problem of active view selection in the context of 3D Gaussian Splatting (3DGS). The central contribution is the derivation of a physically-grounded uncertainty model that operates directly on the explicit parameters of the 3D Gaussian primitives—rather than relying on abstract neural network weights or scene-level metrics. Through a principled propagation of parameter covariance via the rendering Jacobian and subsequent integration of semantic segmentation masks, OUGS isolates object-level uncertainty, enabling targeted camera view selection to maximize object reconstruction quality.

Figure 1: A complex background can inflate image-level uncertainty and mislead active view selection away from the object of interest.

Methodology

Physically-grounded Parameter Uncertainty in 3DGS

Traditional uncertainty estimation approaches in 3D reconstruction often operate at the scene level, leading to suboptimal active view selection especially in scenarios where object fidelity is paramount (Figure 1). OUGS initiates a departure from these methods by explicitly modeling uncertainty for each 3D Gaussian primitive, parameterized by position, scale, rotation, opacity, and Spherical Harmonics for color. These parameters are treated as random variables with an associated covariance, $\Sigma$.

The covariance propagation to image space follows the classic Jacobian–Covariance law. For color $C(u)$ at pixel $u$:

$$\text{Var}[C(u;\theta)] \approx J_u\,\Sigma\,J_u^{\top}$$

where $J_u$ is the pixel-wise Jacobian of color with respect to the full parameter set.

Figure 2: The parameterization of a 3D Gaussian primitive. Each building block is quantified by explicit physical parameters, enabling direct uncertainty estimation.

This pixel-wise uncertainty quantifies how each Gaussian's parameter uncertainty translates into uncertainty in the reconstructed image. The uncertainty can be decomposed into geometric and appearance contributions due to diagonal Fisher Information Matrix (FIM) approximation.

Object-aware Uncertainty via Semantic Masking

To realize object-centric view selection, OUGS incorporates a semantic mask $M_k(u)$ for object $k$, ideally derived from a sufficiently accurate semantic segmentation model. The object-aware uncertainty at pixel $u$ is thus:

$$\Sigma_{C,k}(u) = [M_k(u)]^2 J_u\,\Sigma\,J_u^{\top}$$

By summing over all object pixels, a scalar uncertainty score is obtained that directs active view selection towards regions maximizing the reduction of object uncertainty.

Figure 3: Object-aware uncertainty guides 3DGS view planning for precise object reconstruction, combining physical parameter uncertainty and semantic segmentation.

Fisher Information Matrix-based Covariance Update

Direct computation of full $\Sigma$ is impractical. OUGS adopts a diagonal FIM approximation:

$\Sigma \simeq \sigma^2 \, \mathcal{I}^{-1}$

where $\mathcal{I}$ is tracked online via an exponential moving average (EMA) of squared gradients, with a decaying momentum schedule for robust adaptation:

$$\mathcal{I}_{t,i} = \alpha_t\,\mathcal{I}_{t-1,i} + (1-\alpha_t)\,[\nabla_{\theta_i} \ell_t]^2$$

This update is computationally lightweight and aligns with standard stochastic optimization practices.

Experimental Results

Object-aware Versus Scene-level NBV Selection

Extensive evaluations across Mip-NeRF360, Light-Field, and Tanks & Temples datasets establish the efficacy of OUGS in the active view selection loop. When evaluation is constrained to object masks, OUGS surpasses all prior methods (including FisherRF and GauSS-MI) in PSNR, SSIM, and LPIPS for targeted objects, demonstrating superior view allocation and sharper reconstructions.

Figure 4: Object-aware approach speeds up convergence; OUGS rapidly improves object fidelity while FisherRF lags due to scene-level distraction.

In panoramic (scene-level) evaluations, OUGS remains competitive with information-theoretic approaches, substantiating its general robustness even when global coverage is considered.

Qualitative Analysis

Qualitative results reinforce the quantitative findings. OUGS uniquely maintains high fidelity in object regions, mitigating the misallocations caused by scene-level uncertainty metrics that tend to prioritize background clutter.

Figure 5: OUGS reconstructions preserve object fidelity, outperforming FisherRF and Random policies, which dilute view budget across the background.

Uncertainty heatmaps generated by the method are tightly correlated with actual rendering artifacts. High uncertainty regions predicted by OUGS correspond to blurry or mis-reconstructed areas in the render, demonstrating calibration accuracy.

Figure 6: Uncertainty heatmaps localize prediction errors, confirming physically-grounded model’s reliability in error explanation.

Uncertainty Calibration and Component Analysis

AUSE metrics confirm the FIM-based uncertainty estimator as competitive or superior for error localization compared to variational and ensemble uncertainty methods.

Sensitivity analysis of semantic mask binarization underscores the importance of mask quality in object-aware uncertainty estimation. Over-permissive masks inflate uncertainty scores via background inclusion; overly strict thresholds erode object coverage and degrade performance.

Figure 7: Semantic mask threshold impacts object-level uncertainty calibration; optimal masking isolates the object without discarding informative pixels.

Ablation studies also demonstrate the significance of the EMA schedule. Decaying momentum achieves higher object PSNR than either constant low or high momentum alternatives, affirming the balance between early uncertainty smoothing and late-stage responsiveness.

Implementation and Computational Considerations

OUGS is designed for efficient deployment. Diagonal FIM tracking and Jacobian-based uncertainty propagation require only lightweight per-parameter updates and gradient computations, readily parallelizable on modern hardware. Strong performance is contingent on reliable semantic masking; current implementation utilizes SAM2 for mask generation. The linear per-view computational cost is significantly below that of ensemble or Bayesian 3DGS uncertainty approaches. OUGS integrates with standard NBV planning pipelines by replacing the view selection score with its object-aware uncertainty metric.

Limitations include reliance on mask quality and independence assumptions implicit in diagonal FIM approximation. Extending OUGS to multi-object or global coverage scenarios could require structured FIM variants or composite uncertainty weighting. Scalability for large scenes may require further innovations in online FIM tracking or parameter pruning.

Theoretical and Practical Implications

The explicit modeling of uncertainty on physical parameters aligns uncertainty quantification with the interpretability and controllability desirable in robotics and vision applications. Object-aware uncertainty enables principled NBV selection in object-focused acquisition tasks, advancing the fidelity and efficiency of 3DGS-based reconstruction in real-world scenarios. Methodologically, OUGS sets a precedent for separating geometric and appearance uncertainty, which may inform future uncertainty quantification approaches for other explicit representations.

OUGS opens avenues for further research in self-supervised mask estimation, joint multi-object NBV planning, and adaptive FIM structures for finer uncertainty coupling. The separation of object and background uncertainty also facilitates downstream object-centric tasks in AR/VR, SLAM, and robotic manipulation.

Conclusion

OUGS presents a principled framework for object-centric active reconstruction in 3DGS, advancing the state-of-the-art in uncertainty-driven view planning. By directly propagating physical parameter uncertainty and leveraging semantic masks, OUGS achieves targeted fidelity improvements unattainable by scene-level approaches, with competitive scene-level performance and robust uncertainty calibration. The proposed diagonal FIM and online update strategy provide scalable and interpretable uncertainty estimates, laying groundwork for future object-aware reconstruction methodologies in explicit scene representations.