LIRM: Transformer-Based Volumetric Models

• Transformer-Based Volumetric Models (LIRM) is a deep learning architecture that applies transformer self-attention to fuse multi-view data, reconstruct 3D shapes, and extract relightable assets.
• It employs a progressive update mechanism with token memory and a hexa-plane neural SDF to accurately capture geometric details and spatially-varying material properties.
• The inclusion of a neural directional embedding enables effective modeling of view-dependent effects, achieving state-of-the-art performance in sparse-view inverse rendering.

A Transformer-Based Volumetric Model (LIRM) is a deep learning architecture that applies transformer self-attention mechanisms to the joint reconstruction of 3D shape, spatially-varying material decomposition, and view-dependent volumetric radiance, with the goal of producing high-fidelity inverse rendering from sparse multi-view observations. The LIRM design builds upon and extends the Large Reconstruction Model (LRM) framework, advancing its reconstruction capacity through three key innovations: (1) progressive transformer-based multi-view fusion with token memory for continual assimilation of new observations, (2) a hexa-plane token-based neural signed distance function (SDF) and physically-based material representation, and (3) an integrated neural directional embedding for accurate modeling of complex, glossy and view-dependent appearance, directly enabling relightable asset extraction for graphics engines. Trained on large-scale multi-material datasets, LIRM achieves state-of-the-art shape and radiance field reconstruction in sparse-view regimes, outperforming prior large-scale optimization-based and feed-forward inverse rendering systems, and providing real-time (<1 s/object) relightable mesh and material recovery suitable for production-level 3D content pipelines (Li et al., 28 Apr 2025).

1. Transformer-Based Progressive Multi-View Fusion

The core of LIRM is a deep transformer backbone that fuses visual tokens from multiple input views, maintaining and continually updating a volumetric representation as new observations are introduced. Each input view is provided as a masked object image, background image, and Plücker-ray encoding, which are patch-embedded and concatenated with learnable tri-plane positional tokens. This sequence is processed by a 24-layer, 16-head, 1024-dimension transformer, producing 3D feature plane tokens (Li et al., 28 Apr 2025).

A major departure from prior feed-forward or optimization-based approaches is LIRM's progressive update mechanism. After initial inference, new view sets can be assimilated without reconstructing the entire token memory. Previous output-plane tokens are fused with learnable position encodings, and then refined via self-attention, enabling iterative region-wise refinement as more views arrive: $$P_k^m = \mathrm{MLP}_{\mathrm{fuse}}\bigl([T_k^{m-1}, P_k]\bigr),\qquad \{T_k^m\} = \mathrm{Transformer}(\{P_k^m\},\{I^m\}).$$ This structure achieves memory complexity independent of the number of update steps and empirically converges after relatively few updates (typically 3–4, saturating at ~16 views).

2. Hexa-Plane Token-Based Neural SDF and Material Representation

LIRM adopts a hexa-plane neural tokenization for efficient and expressive volumetric representation. Rather than relying on tri-plane encodings (as in previous LRM models), LIRM uses six axis-aligned half-planes (xy+, xy–, xz+, xz–, yz+, yz–). For any world-space point $\mathbf{x} = (x, y, z)$, bilinear sampling is performed on the relevant positive/negative half-planes: $$\mathbf{f}_{xy}(\mathbf{x}) = \begin{cases} \mathrm{Bilinear}(\mathbf{T}_{xy+}; x, y) & z \ge 0 \ \mathrm{Bilinear}(\mathbf{T}_{xy-}; x, y) & z < 0, \end{cases}$$ with analogous operations for $\mathbf{f}_{xz}$, $\mathbf{f}_{yz}$, yielding a feature vector $\mathbf{f}(\mathbf{x})$.

This feature is input to multiple small MLP decoders to regress SDF $s(\mathbf{x})$, surface normal, diffuse albedo, roughness, metallicity, and color. SDF-based volumetric density $\sigma(s)$ is calculated using a bias-corrected soft step function: $$\sigma(s) = \begin{cases} \frac{1}{2}\exp(-s/\beta), & s \ge 0 \ 1 - \frac{1}{2}\exp(+s/\beta), & s < 0 \end{cases}$$ with $\beta$ annealed through training for coarse-to-fine geometry (Li et al., 28 Apr 2025). This hexa-plane memory arrangement offers improved geometric and material detail recovery, especially for complex or unresolved regions from sparse views.

3. Neural Directional Embedding for View-Dependent Effects

A key innovation in LIRM is its dedicated neural directional embedding (NDE) module for modeling view-dependent appearance. The architecture maintains a set of learnable panoramic feature tensors across update steps, which encode radiance as a function of reflection direction for a given surface point. For an intersection point with normal $\mathbf{x} = (x, y, z)$0 and view direction $\mathbf{x} = (x, y, z)$1, the reflection direction $\mathbf{x} = (x, y, z)$2 is computed, mapped to spherical coordinates $\mathbf{x} = (x, y, z)$3, and a feature token is indexed as: $\mathbf{x} = (x, y, z)$4 where $\mathbf{x} = (x, y, z)$5 is a learned MLP-derived weighting. The final output color is computed as $\mathbf{x} = (x, y, z)$6 (Li et al., 28 Apr 2025). This mechanism enables LIRM to reproduce high-frequency, multi-bounce specularities and relighting effects not accessible to conventional token-based radiance fields.

4. Training Methodology and Inferential Performance

LIRM is trained over 600k physically-based rendered (PBR) 3D models with per-pixel ground-truth albedo, roughness, and normals under 2500 high dynamic range (HDR) environment maps, producing 38 million multiview frames (Li et al., 28 Apr 2025). A three-stage coarse-to-fine curriculum is employed:

• Initial stage with low resolution, few samples/ray, high learning rate.
• Refinement with increased samples/ray, lower learning rate.
• Final stage with maximal samples/ray and strong normal/geometry loss. Core losses include $\mathbf{x} = (x, y, z)$7 on radiance and physically-based properties, LPIPS for perceptual sharpness, SDF-gradient (normal) regularization, and occupancy regularization (Li et al., 28 Apr 2025).

At inference (0.3 s per object on an A100 GPU), LIRM supports mesh extraction via Marching Cubes, baking of spatially-varying PBR texture maps, and environment-mapped radiance coefficient export for graphics engine compatibility.

5. Comparative Evaluation and Relightable Asset Extraction

LIRM achieves state-of-the-art accuracy on geometry, relighting, and material metrics:

• On GSO and ABO benchmarks, it systematically outperforms MeshLRM and GS-LRM in PSNR, SSIM, and LPIPS.
• On Stanford-ORB, LIRM matches or exceeds dense-view optimization-based methdods (e.g., InvRender, NVDiffRecMc, Neural-PBIR) for relighting PSNR/SSIM and Chamfer distance while reducing inference times from hours to sub-second scales (Li et al., 28 Apr 2025).

LIRM supports seamless integration into graphics pipelines. For each vertex $\mathbf{x} = (x, y, z)$8 on the extracted mesh, the MLP decoders can be queried to yield albedo, normal, roughness, metallicity, and optionally an environment-mapped radiance lookup, yielding assets that are directly consumable by physically-based renderers used in modern engines such as Unreal, Unity, or Blender.

6. Limitations and Directions for Extension

Limitations of LIRM include increased memory and computation cost due to the large transformer depth and high token capacity, as well as reliance on substantial training corpora for strong prior-based generalization. The progressive update architecture efficiently amortizes memory, but fine-grained handling of non-local, occluded, or highly reflective regions may require further advances in dynamic memory allocation or finer granularity of directional encoding. A plausible implication suggested by the method design is that such transformer-based models may soon enable production-ready, relightable asset extraction from highly undersampled or real-world image collections, provided further advances in token and memory efficiency.

7. Relationship with Transformer Volumetric Model Landscape

LIRM is situated at the intersection of transformer-based 3D reconstruction, neural rendering, and scene relighting. It is aligned architecturally with recent advances in transformer-based volumetric 3D scene fusion (e.g., VoRTX (Stier et al., 2021)), implicit radiance field modeling (cf. LIRF (Huang et al., 2023)), and physically-based asset recovery, but distinguishes itself through its hexa-plane SDF and material design, progressive fusion, and explicit modeling of view-dependent neural embedding spaces. Its accuracy and memory scaling are enabled by the combination of efficient update modules and representation choices that synergistically exploit transformers' global context modeling (Li et al., 28 Apr 2025).