Invertible Neural Warp for NeRF (2407.12354v1)

Abstract: This paper tackles the simultaneous optimization of pose and Neural Radiance Fields (NeRF). Departing from the conventional practice of using explicit global representations for camera pose, we propose a novel overparameterized representation that models camera poses as learnable rigid warp functions. We establish that modeling the rigid warps must be tightly coupled with constraints and regularization imposed. Specifically, we highlight the critical importance of enforcing invertibility when learning rigid warp functions via neural network and propose the use of an Invertible Neural Network (INN) coupled with a geometry-informed constraint for this purpose. We present results on synthetic and real-world datasets, and demonstrate that our approach outperforms existing baselines in terms of pose estimation and high-fidelity reconstruction due to enhanced optimization convergence.

• The paper presents an INN-based overparameterization method that simultaneously optimizes camera pose and NeRF to boost convergence.
• It leverages invertible neural networks to enforce rigid warp invertibility, ensuring physically plausible transformations.
• Experiments on synthetic and real-world datasets demonstrate significant improvements in pose accuracy and scene reconstruction quality.

An Invertible Neural Network Approach to Pose and NeRF Optimization

The paper "Invertible Neural Warp for NeRF" introduces a novel method for the simultaneous optimization of camera pose and Neural Radiance Fields (NeRF). The authors present an innovative approach that diverges from traditional explicit global representations of camera pose by proposing an overparameterized representation that leverages learnable rigid warp functions. A core innovation in this work is the application of Invertible Neural Networks (INNs) to enforce the critical constraint of invertibility in modeling these rigid warp functions, leading to enhanced optimization convergence and high-fidelity scene reconstruction.

Problem Context and Motivation

NeRF has proven to be a powerful technique for synthesizing photorealistic images from new viewpoints by modeling a volumetric representation of a 3D scene using a multi-layer perceptron (MLP). Despite its efficacy, one of the main challenges NeRF faces is the requirement for precise camera poses, which are often difficult to obtain. Traditional methods such as BARF (Bundle-Adjusting Neural Radiance Fields), NeRFmm, and GARF have been developed to address this challenge by optimizing NeRF and camera poses jointly. However, these methods use a compact parameterization of camera poses which can struggle with poor convergence basins when optimized alongside NeRF.

Proposed Method

The authors aim to enhance the convergence properties by overparameterizing camera poses through INNs. Traditional NeRF setups use extrinsic camera pose parameters to map pixel coordinates and camera centers into viewing rays in a global coordinate system. Instead, this paper proposes overparameterizing the rigid warp function between pixel coordinates and ray space using a neural network, thus potentially improving convergence due to the flexibility offered by overparameterization.

To ensure the parameterized warp functions respect the rigid transformation's invertibility, an INN is employed. The INN architecture allows modeling these functions while maintaining the bijection property, which is crucial for the transform to be valid in both directions (camera to world and world to camera spaces).

Methodology

The paper details the mathematical formulation of the problem and the proposed solution. It reformulates the joint optimization problem, integrating INNs to map from camera to world coordinates and vice-versa through learnable latent codes unique to each frame but a globally shared INN. This allows parameter efficiency and robust pose estimation. The authors also introduce a rigidity prior to ensure the learned transformations conform to physically plausible rigid motions.

Experimental Results

The authors present extensive experimental results across synthetic 2D planar datasets and real-world datasets (LLFF and DTU).

• 2D Planar Neural Image Alignment: The proposed method significantly outperforms traditional MLP-based methods (both naive and implicit-invertible variants) and the baseline method BARF. The results show higher robustness to noise perturbations and better convergence rates, quantified through corner error and patch PSNR metrics.
• Pose and NeRF Optimization: On the LLFF dataset, the proposed method demonstrates a substantial reduction in pose estimation errors compared to BARF and L2G (another recent method leveraging overparameterization through MLP). The improvement in pose accuracy translates directly into better novel view synthesis, with significant improvements in PSNR, SSIM, and LPIPS metrics both before and after test-time photometric optimization.
• 360° Scenes (DTU): The proposed INN-based approach consistently outperforms both BARF and L2G in pose accuracy, with significant improvements in rotation and translation errors. Additionally, geometry evaluations (depth error and Chamfer distance) show that the proposed method achieves more accurate scene reconstructions.

Implications and Future Directions

The findings suggest that overparameterization using invertible neural representations can significantly benefit joint optimization problems in NeRF, offering a promising direction for future research. The work opens several avenues for further exploration:

1. Extension to Dynamic Scenes: The current method could be extended to dynamic scenes where both scene geometry and camera poses need to be optimized over time.
2. Scalability and Real-time Applications: Future work could investigate the scalability of INNs for larger scenes and their applicability in real-time systems.
3. Integration with Other Learning Paradigms: Combining INNs with other learning paradigms, such as reinforcement learning, could yield further improvements in camera pose estimation and scene reconstruction accuracy.

In conclusion, the paper introduces a method that effectively leverages INNs to significantly enhance the robustness and accuracy of joint pose and NeRF optimization. This advancement is critical for applications requiring high-precision scene reconstructions from photometric data and represents a meaningful step forward in neural scene representations.