LiTo: Surface Light Field Tokenization

Abstract: We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.

• The paper presents a novel tokenization mechanism that compresses multiview RGB-D samples into a compact 3D latent space for unified geometry and view-dependent appearance modeling.
• It leverages explicit direction-dependent supervision and higher-order spherical harmonics to faithfully capture specularities, highlights, and Fresnel effects.
• Empirical evaluations show that LiTo outperforms state-of-the-art methods in reconstruction and single-image-to-3D synthesis while reducing latent space size significantly.

LiTo: A Latent Tokenization Framework for Surface Light Fields

Overview and Motivation

LiTo introduces a unified 3D latent representation that encodes both object geometry and view-dependent appearance, specifically targeting the faithful reproduction of effects such as specularities, highlights, and Fresnel reflections. The central contribution is the formulation of a tokenization mechanism for the surface light field, aggregating multiview RGB-D samples into a compressed, structured latent suitable for both faithful reconstruction and efficient single-image-to-3D generation. Unlike previous 3D generative paradigms that decouple geometry and appearance, focus exclusively on view-independent color, or require axis-aligned datamodels that hinder viewpoint alignment, LiTo leverages explicit direction-dependent supervision and flexible continuous input representations.

Figure 1: LiTo tokenizes surface light fields into a latent representation. Reconstructions and single-image-to-3D results demonstrate preservation of geometry and view-dependent effects.

Architecture and Representation

The LiTo pipeline consists of a Perceiver-IO encoder that processes up to one million randomly subsampled surface light field observations for each object, tokenizing them into a compact set of $k=8192$, $d=32$-dimensional latent vectors via cross- and voxel-based localized self-attention (Figure 2). The input includes normalized surface position, color, and local viewing direction per sample, ensuring explicit encoding of angular reflectance variation.

Figure 2: The 3D latent representation compresses subsampled surface light field data into a compact latent, supporting high input cardinality via efficient localized attention.

To decode information, LiTo incorporates specialized Perceiver-based modules: a flow-matching velocity decoder for geometry, a 3D Gaussians decoder using higher-order (up to degree 3) spherical harmonics for view-dependent radiance, an occupancy prediction decoder, and a mesh decoder for downstream mesh recovery needs (Figures 7–9). These decoders enable both differentiable supervision during autoencoding and versatile output modalities at inference.

Patchification in 3D space is achieved via K-nearest neighbor assignment of input samples to randomly selected query positions, enabling cross-attention locality analogously to token patchification in 2D transformers, but for unstructured point clouds.

Training Objectives

The training regime consists of multiple losses designed to ensure that the latent encapsulates both surface geometry and the complete (view-dependent) radiance field:

• Geometry: Flow-matching loss supervises the velocity field over 3D surfaces (modeled as a distribution), enabling both point cloud and normal recovery without explicit mesh requirements.
• Radiance: A multi-view reconstruction loss pushes the latent to capture high-fidelity, direction-dependent appearance, leveraging a 3D Gaussian splatting forward model with higher-degree spherical harmonics for efficient and expressive radiance encoding.

A KL-divergence regularization ensures compact and statistically robust token distributions, aligning posterior latents with a standard normal prior.

Image-Conditioned Generative Modeling

To support single-image-to-3D generation, LiTo trains a latent diffusion model (Diffusion Transformer, DiT) conditioned on DINOv2-derived image embeddings. By canonicalizing the input view orientation to identity during training, LiTo achieves superior alignment between outputs and conditioning images without requiring explicit viewpoint regression or post-hoc alignment, in contrast to systems like TRELLIS.

The generative DiT module incorporates zero-initialized positional encodings and is trained with a flow-matching objective, supported by efficient batch and distributed training on large object datasets.

Empirical Evaluation

Reconstruction

LiTo demonstrates strong numerical performance across multiple challenging datasets and lighting conditions. On Toys4k, GSO, and a PBR-material subset of Objaverse-XL, it substantially improves appearance quality metrics (PSNR, SSIM, LPIPS) and geometric Chamfer distances compared to TRELLIS and other state-of-the-art baseline methods. Notably, LiTo achieves these results with a tenfold reduction in latent space size compared to competing grid-based and primitive field-based models, while requiring less preprocessing and supporting fully continuous inputs. Qualitative results demonstrate clear preservation of view-dependent specular and Fresnel effects under complex illumination (Figure 3).

Figure 3: Reconstructions under varying lighting show accurate recovery of specular and Fresnel reflections.

The generated meshes from LiTo's mesh decoder exhibit higher fidelity and retention of fine-scale geometric detail relative to prior approaches (Figure 4).

Figure 4: Mesh comparison; LiTo recovers finer geometric detail relative to TRELLIS.

Generation

On single-image-to-3D synthesis, LiTo outperforms TRELLIS by a substantial margin in FID, KID, and DINO-based FID on both conditioning and novel views, demonstrating strong fidelity to the reference image and robust generalization to new viewpoints. The coordinate system alignment of generated geometry is demonstrably superior (Figure 5).

Figure 6: Single image to 3D; reconstructions align accurately with the input view while reproducing plausible geometry and appearance.

Figure 5: Input-view fidelity; LiTo’s generations maintain consistent alignment with the conditioning image coordinate system, unlike TRELLIS.

Further, ablations reveal that utilizing higher-order spherical harmonics significantly enhances modeling of view-dependent radiance, while inclusion of viewing direction in encoder inputs furthers effectiveness only when the decoder is sufficiently expressive (Figures 10–11).

Figure 7: Spherical harmonics order studies; higher orders increase view-dependence modeling.

Analysis and Implications

Key claims supported by experiments:

• Compact continuous point-based tokenization enables accurate and disentangled geometry/appearance modeling without reliance on grid-based data structures or mesh preprocessing.
• Incorporation of surface light field directionality (and higher-order SH in the decoder) is essential for high-fidelity, non-diffuse reflectance synthesis and reconstruction.
• Training strategies such as input-view canonicalization are critical for coordinate alignment in image-conditioned generation.

Contrasts with prior art: Unlike GRF- or grid-based representations (e.g., TRELLIS, TripoSG), LiTo supports single-stage generation (no geometry oracle required), fully view-dependent appearance, and flexible alignment with arbitrary input views.

Practical impact: LiTo sets a new standard for the joint modeling of shape and realistic material effects in 3D generative modeling, directly benefiting downstream tasks that require consistent geometry, faithful reflectance, and image-3D alignment. The architecture's ability to scale to millions of tokens (inputs or latent) and modular decoder ecosystem offers extensibility to even more expressive generative paradigms and data types.

Limitations: The current 3DGS implementation restricts SH degree to 3, limiting capture of extremely high-frequency specular or transparent phenomena.

Future Directions

Possible extensions include: supporting arbitrary spherical harmonics degree for higher frequency reflectance, integrating temporal dynamics for 4D scene recovery, adapting the tokenization framework to general scene-level representations, and leveraging the disentangled latent structure for relighting and editing. Advances in one-step generative flows may further accelerate runtime and enable interactive deployment.

Conclusion

LiTo establishes a principled latent tokenization framework for surface light fields, unifying geometry and view-dependent appearance within a scalable, compact, and versatile representation. The system delivers strong empirical advantages in reconstruction fidelity, appearance realism, and generative controllability, marking a significant advance in 3D generative modeling methodology and tools for neural scene understanding and synthesis (2603.11047).