TensoRF: Tensorial Radiance Fields (2203.09517v2)

Abstract: We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).

• The paper introduces TensoRF, a novel approach that leverages tensor decomposition for efficient and compact 3D radiance field modeling.
• It achieves superior rendering quality with PSNR values up to 33.14 while reducing training time from hours to minutes compared to NeRF.
• The method incorporates L1 sparsity and total variation regularization, enabling scalable optimization for practical applications in AR/VR and robotics.

TensoRF: Tensorial Radiance Fields

In the context of 3D scene modeling, radiance fields have emerged as a powerful representation. Traditional methods like NeRF, which utilize MLPs to model radiance fields, have shown substantial promise but suffer from significant training time and memory inefficiencies. The paper "TensoRF: Tensorial Radiance Fields" introduces an innovative approach that leverages tensor decomposition techniques to address these limitations. This essay explores the main contributions, experimental results, and potential implications of TensoRF.

Contribution Overview

The authors propose TensoRF, a novel method for modeling radiance fields by representing the scene as a 4D tensor, corresponding to a 3D voxel grid with multi-channel per-voxel features. The central innovation lies in factorizing this 4D tensor using tensor decomposition techniques, specifically CANDECOMP/PARAFAC (CP) and their novel vector-matrix (VM) decomposition.

Core Components:

1. Tensor Decomposition:
   • CP Decomposition: Factorizes the tensor into a sum of rank-one components characterized by vectors. This decomposition provides a compact and efficient representation.
   • Vector-Matrix (VM) Decomposition: Introduces vector and matrix factors for two modes of the tensor, allowing a more flexible yet still compact representation, enhancing the expressivity compared to CP.
2. Radiance Field Representation:
   • Utilizes the factorized tensors to model per-voxel features and achieves continuous scene representation via trilinear interpolation.
   • Supports both MLP-based and spherical harmonics (SH) based decoding for view-dependent color computation.
3. Efficient Reconstruction:
   • Incorporates L1 sparsity and total variation (TV) regularization to improve reconstruction quality and prevent overfitting.
   • Coarse-to-fine reconstruction allows for efficient and scalable optimization, crucial for high-resolution scene modeling.

Experimental Results

The effectiveness of TensoRF is validated across several datasets, notably Synthetic-NeRF, NSVF, and Tanks and Temples. Key findings include:

1. Rendering Quality:
   • TensoRF consistently outperforms NeRF and several concurrent methods in terms of PSNR and SSIM. The VM decomposition, in particular, achieves superior rendering quality with high PSNR values (up to 33.14 on Synthetic-NeRF).
2. Efficiency:
   • TensoRF requires significantly less memory and computational resources. The reconstruction time is reduced to as low as 10 minutes for the VM decomposition with 192 components, compared to hours or even days required by prior methods.
3. Model Size:
   • The models maintain a compact size, with TensoRF-CP achieving sizes smaller than 4 MB, and the VM variants remaining under 75 MB, significantly smaller than voxel grid-based methods, which can exceed several GB.

Implications and Future Directions

The implications of this research are manifold, both practical and theoretical:

• Practical Applications:
  • The reduced reconstruction time and compact model sizes make TensoRF highly suitable for real-time applications in AR/VR, robotics, and e-commerce. This opens pathways for deploying high-fidelity 3D scene models in resource-constrained environments such as mobile devices.
• Theoretical Contributions:
  • The introduction of tensor decomposition techniques in the context of radiance fields encourages further exploration of low-rank approximations in high-dimensional data.
  • The VM decomposition combines elements of CP and block term decomposition, offering a novel perspective on tensor factorization that could inspire new methodologies in 3D scene representation and beyond.

Future work could explore the integration of TensoRF with dynamic scenes, potentially incorporating temporal dimensions into the tensor framework. Additionally, combining TensoRF with generative models and other neural representations might unlock new capabilities in scene synthesis and editing.

Conclusion

TensoRF presents a significant advancement in the modeling and reconstruction of radiance fields. By employing tensor decomposition, it achieves a balance of rendering quality, efficiency, and compactness that surpasses existing methods. This work not only enhances the practical applicability of radiance field models but also contributes valuable insights into tensor-based representations in computer vision and graphics.