Compressed Light-Field Tokens (CLiFTs)

• Compressed Light-Field Tokens (CLiFTs) are compact, learnable tokens that encode both geometric and appearance information for neural rendering.
• They tokenize multi-view images into high-dimensional features and compress scene details through clustering and neural attention mechanisms.
• CLiFTs enable trade-offs between data size, rendering quality, and processing speed, supporting efficient and adaptive novel view synthesis.

Compressed Light-Field Tokens (CLiFTs) are a class of compact, learnable scene representations in neural rendering that encode both geometric and appearance information from input images as discrete tokens. Each token corresponds to a half-ray, containing localized radiance and geometric descriptors. CLiFTs serve as the fundamental units for compute-efficient novel view synthesis, offering a mechanism for trade-offs in data size, rendering quality, and computational speed by adjusting the number of tokens used at inference. Recent research positions CLiFTs as an innovation that bridges the efficiency of token-based representations with the demands of high-fidelity, adaptive neural rendering (Wang et al., 11 Jul 2025).

1. Principle and Motivation

CLiFTs are motivated by the need to represent complex 3D scenes with significantly reduced data size while retaining crucial cues for high-quality rendering. Traditional rendering pipelines—such as volumetric reconstruction or multi-plane image interpolation—often operate on dense, pixel- or ray-level data, incurring high memory and computation costs. In contrast, CLiFTs achieve efficient compression by:

• Tokenizing multi-view image data paired with camera poses, transforming per-pixel information (including 3D ray geometry and RGB appearance) into high-dimensional feature tokens.
• Condensing these tokens through clustering and neural attention mechanisms, maintaining relevant visual and geometric context.
• Enabling variable compute: the number of tokens leveraged for rendering can be dynamically chosen based on hardware constraints or user requirements, supporting adaptive real-time applications and scalable scene storage.

CLiFTs are designed to preserve the fidelity of both appearance (color, texture) and geometry (via 6D Plücker coordinates), making them suitable for advanced view synthesis tasks.

2. Scene Tokenization and Latent-Space Clustering

The CLiFT construction process initiates with a multi-view encoder (e.g., Transformer-based) that receives as input a set of images and their corresponding camera poses. The procedure includes:

• Patchification: Each input image is divided into non-overlapping patches (e.g., 8×8 pixel blocks).
• Token Feature Construction: For every patch, a feature vector is formed by concatenating 6D Plücker coordinates (capturing ray direction and origin) with normalized RGB values, resulting in a feature map (e.g., 576 dimensions) that is linearly projected to a higher-dimensional token space (e.g., 768 dimensions).
• Cross-View Aggregation: All patches from all input images are processed to generate a dense collection of tokens (LiFTs), each implicitly representing a half-ray traversing the scene from a particular camera.

To reduce redundancy, especially in regions covered by multiple views or with homogeneous appearance, the system performs clustering in the latent token space:

• Latent-space K-means: Tokens are clustered based on their semantic and geometric similarity. The centroid of each cluster, found as the nearest actual token to the mean, is selected as a “storage token.”
• Coverage Density: This process ensures that textured or geometrically complex regions are given more representative tokens, whereas uniform or redundant regions are compressed more aggressively.

By setting the number of clusters (Nₛ), the desired trade-off between storage size and anticipated rendering quality is directly controlled.

3. Neural Condensation and Information Compression

After clustering, a multi-view “condenser,” implemented as a lightweight Transformer decoder, further compresses and enriches the representative (centroid) tokens:

• Self-Attention: Centroid tokens interact with one another, establishing global consistency across the selected set.
• Cross-Attention: Each centroid token acts as a query attending over the tokens within its assigned cluster, gathering pertinent details from the local region.
• Feed-Forward Network (FFN) and Residual Updates: The aggregated information is processed through FFN layers and combined back with the centroid token via a residual connection (optionally initialized to zero for stability).

Two such condensation blocks are typically stacked, culminating in a set of CLiFTs that collectively encapsulate the scene’s visual and geometric attributes.

4. Compute-Adaptive Rendering and Novel View Synthesis

At inference, the CLiFT-based rendering pipeline accepts a target camera pose and a user-specified “compute budget” (Nᵣ), which dictates the number of tokens engaged for that specific novel view:

• Token Selection Algorithm: The target view is subdivided into patches (e.g., a 24×24 grid). For each patch, the algorithm computes distances (based on ray geometry and camera positions) between the patch’s center ray and the available CLiFTs, greedily selecting the closest tokens up to Nᵣ unique selections. This ensures relevant coverage across the image plane.
• Neural Renderer: A Transformer decoder takes patchified query tokens (from the target view) and cross-attends to the selected CLiFTs. The architecture typically consists of several stacked decoder blocks, each containing self-attention (among queries), cross-attention (between queries and CLiFTs), and FFN layers.
• RGB Projection and Assembly: The renderer outputs are mapped to RGB values (via sigmoid activation) and unpatchified to form the synthesized high-resolution image.

This compute-adaptive architecture allows application scenarios ranging from low-latency, low-resource preview modes (using few tokens), to high-fidelity, compute-intensive renders (using a larger selection of CLiFTs).

5. Performance and Trade-Offs

Extensive quantitative and qualitative validation on datasets such as RealEstate10K and DL3DV demonstrates:

• Data Reduction: CLiFTs provide up to 5–7× data size reduction over reconstruction-based methods (e.g., MVSplat, DepthSplat) and about 1.8× over reconstruction-free baselines (e.g., LVSM), while maintaining or surpassing rendering quality as measured by PSNR, SSIM, and LPIPS.
• Quality-Compute Scalability: Rendering with fewer tokens significantly accelerates frame rates and reduces floating point operations (FLOPs), while maintaining visual quality with only modest degradation. Increasing token count incrementally improves image fidelity.
• Visual Fidelity: Even at high compression, CLiFTs preserve sharp appearance details and accurate geometry with only limited blurring or loss of high frequencies in extreme cases.
• Flexibility: The system’s capacity for fine-grained trade-offs among data size, image quality, and rendering speed is particularly advantageous for interactive and real-time rendering contexts.

The following table summarizes key trade-offs:

# Render Tokens (Nᵣ)	Relative Data Size	Rendering Quality (PSNR)	Render Speed (FPS)
Small	Minimal	Lower	Highest
Medium	Moderate	Good	Moderate
Large	Highest	Best	Lowest

6. Technical Innovations and Implementation Details

Key architectural features and operational details include:

• 6D Plücker Ray Encoding: Ensures that each token represents both spatial direction and origin, supporting accurate geometric scene encoding.
• Patchification and High-Dimensional Projection: Facilitates efficient scene tokenization, balancing granularity with computational feasibility.
• Heuristic Token Selection: Employs a simple yet effective approach for identifying relevant tokens for novel views, considering ray distance and camera proximity.
• Compute-Adaptive Design: Trains the neural renderer to handle dynamically varying numbers of context tokens, promoting robustness and scalability without retraining.

The encoding pipeline can be characterized by the following stages and operations:

• Input images + camera poses → patchify + concatenate geometry/color (feature vectors) → linear projection → multi-view encoding → latent K-means clustering → condensation Transformer decoder → CLiFTs.

The rendering pipeline involves:

• Target pose + grid → token selection → query tokenization → Transformer decoder (self- & cross-attention, FFN) → RGB output → unpatchify.

7. Applications and Prospects

CLiFTs enable scalable, storage-efficient, and real-time friendly neural rendering, directly supporting use cases such as:

• Arbitrary viewpoint synthesis for immersive 3D viewing, virtual reality, or machine learning systems requiring compact but expressive scene features.
• Resource-constrained inference on embedded or mobile devices, thanks to adjustable token counts and efficient decoding.
• Large-scale scene storage or streaming with adaptive quality based on bandwidth or interactivity constraints.
• Integration with more advanced scene understanding or editing pipelines, given their compact representation and geometric interpretability.

Emerging research directions include expanding CLiFT constructions for temporal (video) domains, developing more adaptive or content-aware clustering schemes, and integrating generative rendering or compositional editing functionalities.

---

In summary, compressed light-field tokens (CLiFTs) represent a principled, token-based approach for encoding, storing, and rendering 3D scenes. They provide explicit control over the trade-off between computational demand and rendering fidelity, facilitating efficient scene representation while preserving essential appearance and geometric detail. CLiFT-based methods are validated as state-of-the-art in data reduction and rendering efficiency on contemporary neural rendering benchmarks (Wang et al., 11 Jul 2025).