Is Attention All That NeRF Needs? (2207.13298v3)

Abstract: We present Generalizable NeRF Transformer (GNT), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views. While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages. (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinate-aligned features by aggregating information from epipolar lines on the neighboring views. (2) The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching. Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer. When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~10% on average. Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physically-grounded rendering. Our results show the promise of transformers as a universal modeling tool for graphics. Please refer to our project page for video results: https://vita-group.github.io/GNT/.

• The paper introduces a two-stage transformer architecture that uses view and ray transformers to reconstruct and render neural radiance fields efficiently.
• The paper achieves state-of-the-art performance with an average 10% improvement over traditional methods in challenging lighting and occlusion scenarios.
• The paper demonstrates that generalizable attention mechanisms can replace handcrafted rendering equations, enabling scalable and robust neural rendering.

Generalizable NeRF Transformer (GNT) for Neural Rendering

The paper "Is Attention All That NeRF Needs?" introduces the Generalizable NeRF Transformer (GNT), a novel transformer-based approach designed to reconstruct Neural Radiance Fields (NeRFs) from source views and render novel perspectives. Unlike traditional NeRF methods that require optimizing a scene representation using a handcrafted rendering equation, GNT leverages a two-stage transformer architecture to enable efficient neural representation and rendering. This approach facilitates generalization across different scenes.

Methodology

The core contribution of the paper lies in its two-stage transformer framework:

1. View Transformer: This network component utilizes multi-view geometry as an inductive bias to enhance attention-based scene representation. By consolidating information from epipolar lines across neighboring views, it predicts coordinate-aligned features. This approach extends beyond conventional methods that rely heavily on scene-specific optimizations, thereby improving cross-scene adaptability.
2. Ray Transformer: This module renders novel views by decoding features along sampled points during ray marching through attention mechanisms. By utilizing self-attention, the ray transformer aggregates feature representations without resorting to explicit volume rendering equations.

Experimental Results

The experimental setup involved both single-scene optimizations and evaluations across multiple scenes for generalization. GNT demonstrated robust performance, especially in scenarios involving complex lighting conditions and occlusions. When trained on multiple scenes, the transformer architecture consistently achieved state-of-the-art results, outperforming other methods by approximately 10% on average. These results underscore the effectiveness of transformer-based architectures in rendering high-fidelity images from neural radiance fields.

Implications and Future Directions

The research highlights the potential of transformers as a universal modeling tool for graphics, suggesting that explicit scene representation and hard-coded rendering equations used in traditional methods can be substituted with a more generalizable attention mechanism. This provides a significant advantage in handling complex scenes with diverse geometries and lighting conditions without specific tuning.

From a practical standpoint, replacing dense computational processes with attention-based approaches can lead to more efficient and scalable neural rendering systems. Theoretically, this paves the way for using transformers in various domains beyond graphics, potentially influencing how neural networks model and interpret 3D data.

Future research may explore relaxing or modifying the inductive biases related to epipolar geometry to better simulate complex light transport phenomena. The paper further suggests potential extensions, such as auto-regressive rendering and attention-based coarse-to-fine sampling, which could improve rendering quality and computational efficiency.

Conclusion

The GNT framework represents a significant step in applying transformer architectures to neural rendering, offering insights into both practical advancements in rendering applications and theoretical understanding of attention mechanisms in modeling complex visual data. Its ability to generalize across scenes and render intricate details without relying heavily on handcrafted modeling equations marks a promising direction for future research.