RenderFormer++: Scalable and Physically Grounded Feed-Forward Neural Rendering

Abstract: We present RenderFormer++, a scalable and physically grounded feed-forward neural rendering framework for global illumination in mesh scenes. Existing Transformer-based neural rendering methods such as RenderFormer achieve promising cross-scene generalization, but suffer from limited physical consistency and poor scalability due to the quadratic attention complexity of triangle-level tokenization. To address these issues, we introduce Physics-Informed Transport Guidance (PITG), which embeds rendering-equation inductive biases into the attention mechanism and enforces transport consistency loss, enabling physically consistent light transport modeling. We further propose Hierarchical Object-Centric Tokenization (HOCT), which aggregates triangle-level features into compact object-level tokens via cross-attention with learnable queries, substantially reducing computational and memory costs while preserving geometric and radiometric information. Extensive experiments demonstrate that RenderFormer++ achieves scalable, stable, and generalizable feed-forward global illumination rendering across complex large-scale scenes with improved physical accuracy and efficiency over prior neural rendering methods.

• The paper presents a novel feed-forward neural rendering framework that integrates Physics-Informed Transport Guidance (PITG) to embed explicit priors from the rendering equation.
• It introduces Hierarchical Object-Centric Tokenization (HOCT) to convert triangle-level data into compact object-level tokens, enabling linear scalability in memory and runtime.
• The method achieves superior global illumination results with enhanced physical accuracy and efficiency, validated by both qualitative and quantitative experiments on complex scenes.

RenderFormer++: Scalable and Physically Grounded Feed-Forward Neural Rendering

Introduction

RenderFormer++ advances the state of neural global illumination by addressing two principal deficiencies in prior transformer-based rendering frameworks: their inadequate physical grounding and their poor scalability due to the quadratic complexity of triangle-level attention. Transformer-based neural rendering models such as RenderFormer have demonstrated cross-scene generalization by modeling triangle-level interactions with self-attention, but they suffer from limited scalability when faced with high geometric complexity and fail to enforce the physical principles embedded in the rendering equation. RenderFormer++ remedies these limitations by introducing Physics-Informed Transport Guidance (PITG) and Hierarchical Object-Centric Tokenization (HOCT), resulting in a highly scalable, efficiently trainable, and physically consistent feed-forward renderer for complex triangle-mesh environments.

Physics-Informed Transport Guidance (PITG)

RenderFormer++ incorporates explicit priors from the rendering equation into the attention-based neural architecture to bridge the gap between data-driven modeling and physical accuracy. PITG introduces the following mechanisms:

1. Emission Encoder: Emission priors are directly injected by encoding the location and characteristics of emissive surfaces, allowing the network to mimic the source-term structure of the rendering equation.
2. Neural Light Transport Operator: The transport process is recursively modeled by a transformer-based operator applied iteratively to account for multi-bounce light propagation. This operator explicitly segregates emitted and transported radiance, ensuring that recursive light interactions are captured in the feature space.
3. Transport Consistency Loss: A regularization objective penalizes deviations between successive transport iterations, encouraging stable and physically consistent convergence toward global illumination solutions in the neural feature domain.

This modification directly introduces rendering-equation-induced biases into the transformer, improving the physical plausibility of the resultant illumination while retaining efficient feed-forward inference.

Hierarchical Object-Centric Tokenization (HOCT)

Scalability in direct-attention architectures is typically constrained by quadratic complexity in the number of scene primitives. RenderFormer++ resolves this through HOCT, which:

• Aggregates triangle-level tokens into compact object-level tokens via cross-attention with learnable queries, significantly reducing the token sequence length processed by attention layers.
• Preserves the essential geometric and radiometric context by using sophisticated pooling and positional encoding strategies.
• Enables linear scaling in both memory and runtime with respect to the number of objects, as empirically validated by extensive profiling.

Under HOCT, even large and complex mesh scenes are tractable, and memory usage remains manageable for practical hardware across a wide range of scene complexities.

Figure 1: Scalability analysis of RenderFormer++ demonstrating its significantly improved memory and runtime efficiency as scene complexity grows, relative to triangle-level tokenization baselines.

Geometry-Guided Decoding

After global object-level aggregation and physically informed transport modeling, RenderFormer++ decodes view-dependent radiance using geometry-guided tokens:

• Primary ray bundles for image patches are enhanced with G-buffer attributes (normals, albedo, and roughness), granting localized information crucial for high-fidelity appearance synthesis.
• A dedicated transformer-based decoder fuses these geometry-aware patch queries with global illumination tokens, supporting accurate reproduction of view-dependent and fine-grained lighting effects.

Quantitative and Qualitative Results

On both small- and large-scale synthetic datasets, RenderFormer++ attains superior accuracy and perceptual scores (L1, MAPE, LPIPS, SSIM) compared to RenderFormer—particularly marked on scenes with higher geometric complexity, where the baseline model fails due to computational infeasibility or degraded accuracy.

Visually, RenderFormer++ produces robust indirect illumination, soft shadowing, color bleeding, and specular interreflections over unseen complex scenes, evidencing generalization beyond the training distribution.

Figure 2: Qualitative results on challenging unseen mesh scenes, with error heatmaps corroborating RenderFormer++'s fidelity and physical consistency in global illumination.

Ablation Studies

Systematic ablation demonstrates the functional contribution of each RenderFormer++ innovation:

• PITG alone substantially boosts all reconstruction and perceptual metrics, both in triangle-level and object-level tokenization regimes, by enforcing structured transport modeling.
• Transport Consistency Loss (TCL) stabilizes training, mitigating artifacts due to non-convergent recursive updates.
• HOCT is the critical enabler of large-scene tractability, without which model training fails on high-complexity scenes due to resource exhaustion.
• Geometry-Guided Decoding increases fine-detail fidelity, especially in challenging lighting conditions.
• Increasing the number of transport iterations ($K$) further enhances global illumination accuracy at the cost of moderate extra computation, with practical trade-offs for $K=3$.

Implications, Limitations, and Future Directions

RenderFormer++ establishes a scalable paradigm for transformer-based physically grounded neural rendering, offering the practicality and efficiency required to handle large, real-world mesh scenes in global illumination tasks. The use of HOCT and PITG is generalizable beyond triangle meshes and may inform future architectures for other spatially structured neural scene representations.

However, several simplifying assumptions—such as uniform material within each object and omission of complex textures—restrict immediate deployment to textured and highly heterogeneous scenes. Preliminary results on basic textured scenarios suggest compatibility, and future work should extend encoding and tokenization mechanisms to fully support arbitrarily textured, material-diverse objects. Further integration of higher-order transport mechanisms and material-specific priors may bridge remaining quality gaps with traditional Monte Carlo path tracing in the most challenging settings.

(Figure 3)

Figure 3: Preliminary evaluation of RenderFormer++ on textured scenes, indicating extensibility to richer material and appearance variation.

Conclusion

RenderFormer++ constitutes a significant step in scaling feed-forward neural global illumination to complex triangle-mesh environments, while remaining faithful to the physical structure of the rendering equation. Through the principled design of object-centric representations and recursive, physically informed transport modeling, it achieves a blend of quality, generalization, and computational tractability not previously attained in transformer-based neural rendering. Its methodologies lay the groundwork for future large-scale, physically accurate neural approaches to light transport modeling in computer graphics and vision.