RenderFormer++: Scalable Neural Rendering

• The paper introduces a scalable feed-forward neural rendering framework that predicts converged HDR images in one pass, eliminating per-scene optimization and Monte Carlo sampling.
• It employs hierarchical object-centric tokenization and physics-informed transport guidance to overcome quadratic attention bottlenecks and ensure transport consistency.
• Empirical evaluations demonstrate superior perceptual quality and stability on scenes with 10K–80K triangles compared to its predecessor, RenderFormer.

RenderFormer++ is a scalable and physically grounded feed-forward neural rendering framework for global illumination in mesh scenes. It is designed to predict a converged HDR rendering from an explicit triangle-mesh scene and a camera pose in one forward pass, without per-scene optimization or Monte Carlo sampling at test time. The method extends RenderFormer by addressing two limitations identified in the earlier Transformer-based formulation: limited physical consistency and poor scalability caused by the quadratic attention complexity of triangle-level tokenization (Du et al., 29 Jun 2026).

1. Origins and problem setting

RenderFormer++ addresses cross-scene neural rendering of global illumination for explicit triangle meshes. The target effects include indirect illumination, soft shadows, color bleeding, and specular interreflections. Its intended regime is simultaneously accurate, efficient, scalable, and general across unseen scenes. In the paper’s formulation, rendering is amortized as a learned feed-forward map

$\mathbf{I} = f_\theta(\mathcal{S}, \mathcal{C}),$

where $\mathcal{S}$ is the scene and $\mathcal{C}$ is the camera pose (Du et al., 29 Jun 2026).

The immediate precursor is RenderFormer, which formulated rendering as a sequence-to-sequence transformation from triangle tokens to pixel-patch tokens. RenderFormer used a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the view-independent stage. That formulation established cross-scene rendering from triangle meshes without per-scene training or fine-tuning, but it also exposed a central bottleneck: the view-independent stage scales roughly as $\mathcal{O}(\#tris^2)$, and the practical training limit was $4{,}096$ triangles (Zeng et al., 28 May 2025).

RenderFormer++ preserves the feed-forward and cross-scene character of the original system, but reworks both its representation and its inductive biases. The result is a renderer that remains Transformer-based while becoming more explicitly transport-aware and more tractable for scenes with many objects and many triangles.

2. Scene formulation and rendering model

The scene is represented as a collection of triangles,

$\mathcal{S}=\{t^m\}_{m=1}^{M},$

with each triangle carrying spatial coordinates, shading normals, surface reflectance properties, and emission profiles. The reflectance model is explicitly a microfacet BRDF with GGX NDF, parameterized by diffuse albedo, specular albedo, and roughness. The same scene can also be reorganized as a set of objects,

$$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$

which is the representation used by the hierarchical tokenization scheme (Du et al., 29 Jun 2026).

The rendering target is a converged HDR image

$\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$

During decoding, the image is partitioned into $4\times 4$ patches, and each patch is associated with a bundle of primary rays through the centers of its constituent pixels. The decoder also uses a pre-rendered G-buffer containing attributes such as surface normals, albedo, and roughness (Du et al., 29 Jun 2026).

The physical motivation is framed through the rendering-equation-inspired transport form

$\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$

where $\mathcal{S}$0 denotes outgoing radiance, $\mathcal{S}$1 emitted radiance, and $\mathcal{S}$2 transported and scattered radiance. Defining

$\mathcal{S}$3

gives

$\mathcal{S}$4

and the recursive transport form

$\mathcal{S}$5

This decomposition is central to RenderFormer++: it does not analytically solve the rendering equation, but it organizes its feature space and training objective around emitted radiance, recursive transport, and transport consistency (Du et al., 29 Jun 2026).

3. Architectural organization

RenderFormer++ is a two-stage feed-forward architecture. The view-independent stage comprises triangle embedding, object-centric tokenization, and physics-informed transport modeling; the view-dependent stage performs geometry-guided decoding from ray/G-buffer patch queries (Du et al., 29 Jun 2026).

In the first step, each object’s triangles are encoded as an unordered set. Triangle features include spatial coordinates encoded by Relative Spatial Positional Encoding (RSPE), shading normals, emission profiles, and BRDF parameters. This produces encoded triangle tokens $\mathcal{S}$6. The primitive-level encoder is shared across objects.

The second step is Hierarchical Object-Centric Tokenization (HOCT). A shared learnable latent query set

$\mathcal{S}$7

aggregates each object’s variable-length triangle token set into a fixed-length object representation: $\mathcal{S}$8 All object token sets are then concatenated: $\mathcal{S}$9 The implemented configuration uses $\mathcal{C}$0 learnable object queries, a 3-layer cross-attention module, and a 2-layer self-attention Transformer on object-level tokens.

The third and fourth steps implement the transport model. RenderFormer++ separates emission-related and non-emission-related features through an emission encoder: $\mathcal{C}$1

$\mathcal{C}$2

A Transformer-based neural transport operator then recursively propagates illumination in feature space: $\mathcal{C}$3 with shared-parameter multi-bounce refinement

$\mathcal{C}$4

The final model uses $\mathcal{C}$5 transport iterations and forms

$\mathcal{C}$6

The final step is the geometry-guided decoder. The target image is split into $\mathcal{C}$7 patches. For each patch, primary rays are encoded, corresponding G-buffer attributes are extracted, and these are concatenated into a geometry-guided patch token $\mathcal{C}$8. A Transformer-based decoder then predicts the HDR image: $\mathcal{C}$9 The architecture details are explicit: a 4-layer self-attention Transformer for triangle encoding, a 2-layer Transformer as neural transport operator, and a 6-layer Transformer plus DPT head for the geometry-guided decoder (Du et al., 29 Jun 2026).

4. Physics-Informed Transport Guidance

Physics-Informed Transport Guidance (PITG) is the main mechanism by which RenderFormer++ becomes physically grounded. PITG consists of three parts: an emission encoder, a neural light transport operator, and a transport consistency loss. Its purpose is to replace a largely unconstrained attention-based transport model with one that structurally reflects source radiance initialization, recursive transport dynamics, and stable transport updates (Du et al., 29 Jun 2026).

The first physical prior is explicit separation between emitted radiance and transported or scattered radiance. The feature pairs $\mathcal{O}(\#tris^2)$0 and $\mathcal{O}(\#tris^2)$1 mirror the decomposition $\mathcal{O}(\#tris^2)$2. The second prior is recursive transport, implemented by repeated application of the shared neural transport operator $\mathcal{O}(\#tris^2)$3. The third prior is regularization of transport dynamics through the transport consistency loss: $\mathcal{O}(\#tris^2)$4

The full training objective combines $\mathcal{O}(\#tris^2)$5 loss in log-transformed HDR space, LPIPS loss, and transport consistency loss: $\mathcal{O}(\#tris^2)$6 with

$\mathcal{O}(\#tris^2)$7

A common misconception is to interpret PITG as an analytic insertion of rendering-equation factors directly into attention. The paper does not provide a modified dot-product attention formula with explicit BRDF, visibility, or geometry factors in the attention logits. It also does not specify any explicit modification to queries, keys, values, attention weights, or analytic transport matrices. The physical grounding is therefore structural and supervisory rather than an explicit analytic transport discretization. This is reinforced by the paper’s statement that it does not provide explicit formulas for visibility terms, geometry factors, BRDF kernels inside attention, or energy conservation constraints (Du et al., 29 Jun 2026).

5. Hierarchical Object-Centric Tokenization and scalability

HOCT is the principal scalability mechanism in RenderFormer++. It replaces flat triangle tokenization with a hierarchical object-centric representation, so transport attention operates on compact object-level tokens rather than on all triangles directly. If each object receives $\mathcal{O}(\#tris^2)$8 latent tokens, then the global sequence length becomes roughly $\mathcal{O}(\#tris^2)$9 rather than $4{,}096$0. Since the original RenderFormer uses triangle-level tokenization and full self-attention over all triangle tokens, HOCT directly attacks the quadratic bottleneck (Du et al., 29 Jun 2026).

The method is motivated by large scenes with roughly $4{,}096$1 triangles. In that regime, the paper explicitly states that RenderFormer cannot be trained on the large-scale dataset due to memory overhead from triangle-level tokenization. By contrast, RenderFormer++ can be trained on scenes with $4{,}096$2 triangles due to HOCT. A controlled scaling analysis increases the number of objects from $4{,}096$3 to $4{,}096$4, each with $4{,}096$5 triangles, and shows that RenderFormer++ has much slower growth in peak GPU memory and per-step runtime, although the text does not print exact numeric curve values (Du et al., 29 Jun 2026).

The contrast with the original RenderFormer is therefore architectural rather than merely incremental. RenderFormer’s formulation is permutation-insensitive with respect to triangle ordering and powerful for scene-wide interaction modeling, but its training scenes support up to $4{,}096$6 triangles during pretraining and $4{,}096$7 triangles during fine-tuning, with graceful degradation beyond that range at inference time (Zeng et al., 28 May 2025). RenderFormer++ turns that flat triangle-token design into a hierarchy-aware renderer by preserving detailed triangle embeddings locally while compressing them into object-level transport tokens globally.

6. Training setup, datasets, and empirical results

RenderFormer++ is trained on a synthetic large-scale dataset of $4{,}096$8 scenes following the RenderFormer data generation pipeline. Each scene is procedurally generated by randomizing object instances, materials, lighting, and camera viewpoints. The dataset contains $4{,}096$9 to $\mathcal{S}=\{t^m\}_{m=1}^{M},$0 objects per scene, each object has about $\mathcal{S}=\{t^m\}_{m=1}^{M},$1 triangles, each scene is rendered from $\mathcal{S}=\{t^m\}_{m=1}^{M},$2 viewpoints, and the total training set contains $\mathcal{S}=\{t^m\}_{m=1}^{M},$3 training images. Two subsets are defined: a small-scale dataset of $\mathcal{S}=\{t^m\}_{m=1}^{M},$4 scenes with about $\mathcal{S}=\{t^m\}_{m=1}^{M},$5 triangles per scene, and a large-scale dataset of $\mathcal{S}=\{t^m\}_{m=1}^{M},$6 scenes with about $\mathcal{S}=\{t^m\}_{m=1}^{M},$7 triangles per scene. Reference images are rendered in Blender at $\mathcal{S}=\{t^m\}_{m=1}^{M},$8 resolution with adaptive sampling, $\mathcal{S}=\{t^m\}_{m=1}^{M},$9 spp, and denoising (Du et al., 29 Jun 2026).

Optimization uses AdamW, warmup to $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$0 over the first $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$1 steps and then cosine decay, bfloat16 precision, Flash Attention, batch size $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$2, and $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$3 iterations on $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$4 NVIDIA A100 $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$5GB GPUs for $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$6 days. Evaluation includes a train split set of $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$7 small-scale scenes plus $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$8 large-scale scenes sampled from the training distribution, and a test split set of $$\mathcal{S}=\{O_n\}_{n=1}^{N}, \qquad O_n = \{t_n^m\}_{m=1}^{M_n},$$9 small-scale scenes plus $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$0 large-scale scenes that are completely unseen during training (Du et al., 29 Jun 2026).

On the small-scale dataset, RenderFormer++ achieves $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$1 L1, $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$2 MAPE, $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$3 LPIPS, and $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$4 SSIM on the train split, and $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$5, $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$6, $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$7, and $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$8 on the test split. RenderFormer records $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$9, $4\times 4$0, $4\times 4$1, and $4\times 4$2 on the train split, and $4\times 4$3, $4\times 4$4, $4\times 4$5, and $4\times 4$6 on the test split. RenderFormer (finetune) obtains lower L1 and MAPE than RenderFormer++, but substantially worse LPIPS and SSIM, which the paper interprets as inferior perceptual quality despite competitive pixelwise error (Du et al., 29 Jun 2026).

The large-scale regime is the decisive setting. On the train split, RenderFormer++ reports $4\times 4$7 L1, $4\times 4$8 MAPE, $4\times 4$9 LPIPS, and $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$0 SSIM, compared with $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$1, $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$2, $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$3, and $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$4 for RenderFormer. On the test split, RenderFormer++ reports $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$5, $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$6, $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$7, and $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$8, compared with $\mathbf{L}_o = T\mathbf{L}_o + \mathbf{L}_e,$9, $\mathcal{S}$00, $\mathcal{S}$01, and $\mathcal{S}$02 for RenderFormer. These results support the paper’s claim that RenderFormer++ remains stable and generalizable in scene scales where RenderFormer effectively breaks down (Du et al., 29 Jun 2026).

Ablations isolate the roles of PITG, TCL, the geometry-guided decoder, and transport depth. In the triangle-level setting, integrating PITG into RenderFormer improves most metrics relative to RenderFormer (finetune). In the object-level large-scale setup, removing PITG or TCL degrades reconstruction quality, while removing the geometry-guided decoder causes the largest drop. For transport iteration depth, $\mathcal{S}$03 is slightly better than $\mathcal{S}$04, and $\mathcal{S}$05 provides little additional gain, leading to the final choice $\mathcal{S}$06 (Du et al., 29 Jun 2026).

7. Limitations, interpretation, and significance

RenderFormer++ is physically grounded in the sense of architectural priors and transport-consistency supervision, but it is not an analytic renderer. The paper explicitly notes several omissions from the physical model: no explicit formulas for visibility terms, geometry factors, BRDF kernels inside attention, analytic transport matrices, or energy conservation constraints. Its physical consistency is therefore stronger than that of a purely data-driven attention model, yet still approximate in the classical rendering sense (Du et al., 29 Jun 2026).

The most explicit limitation concerns data and material richness. The main dataset does not include textured objects, and material properties are shared across triangles within each object. The paper mentions preliminary experiments on simple textured scenes suggesting compatibility with richer appearance, but does not treat that as a validated result. It also does not provide a detailed failure-case taxonomy for thin geometry failures, highly glossy caustics, extreme HDR lighting, or disocclusion artifacts (Du et al., 29 Jun 2026).

Within neural rendering, RenderFormer++ occupies a specific position. Relative to scene-specific neural transport or neural radiosity-like methods, it remains cross-scene and feed-forward. Relative to the original RenderFormer, it introduces rendering-equation-inspired structure through emission/source disentanglement, recursive transport, and transport consistency loss, while also replacing flat triangle tokenization with a hierarchy-aware object-centric representation (Zeng et al., 28 May 2025). Conceptually, this turns the earlier generic attention-based renderer into a more structured transport-aware and hierarchy-aware renderer.

From a systems perspective, the architecture remains Transformer-based rather than U-Net-like. This matters for subsequent acceleration research: ReFrame, a training-free inference acceleration method based on caching intermediate activations, is explicitly designed for U-Net and encoder-decoder style networks only and “does not support other architectures such as transformers” (Liu et al., 14 Jun 2025). A plausible implication is that RenderFormer++ establishes a strong rendering formulation, but efficient temporal reuse for this class of Transformer renderers remains an open engineering direction rather than a solved component of the published method.

RenderFormer++ is therefore best understood as a cross-scene feed-forward global illumination renderer for triangle-mesh scenes whose main advances are orthogonal and complementary: stronger physical structure through PITG, and large-scene scalability through HOCT. Its significance lies less in analytic fidelity to the rendering equation than in demonstrating that Transformer-based neural rendering can be made simultaneously transport-aware, hierarchy-aware, and practical for scenes with $\mathcal{S}$07 triangles (Du et al., 29 Jun 2026).