RGBX-Next: Towards Realistic Generative Rendering from G-Buffers

Abstract: Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.

• The paper introduces a unified video-diffusion framework that converts between RGB sequences and G-buffers, using frame-wise tokenization, QK type embeddings, clean input tokens, and X-patchify to distinguish modalities and diffusion roles.
• The paper demonstrates strong inverse rendering, including Hypersim depth results of 29.47 PSNR and 0.9032 SSIM, while real-video training reduces forward-rendering FID to 45.3871 compared with 62.3884 for RGB↔X.
• The paper shows that estimated G-buffers, text guidance, lighting buffers, condition dropout, and hybrid teacher/self-forcing enable controllable realistic rendering and temporally stable streaming generation for approximately 1,000 frames, though inference remains computationally expensive.

Problem formulation and contribution

“RGBX-Next: Towards Realistic Generative Rendering from G-Buffers” (2608.13929) addresses the controllability gap between physically based rendering and diffusion-based image and video synthesis. Traditional rendering provides explicit, spatially and temporally stable control over geometry, materials, illumination, and motion, but its visual realism depends on expensive scene authoring and high-quality assets. Diffusion models provide strong generative priors and realistic appearance, but conventional conditioning mechanisms generally offer weaker guarantees about scene structure and temporal consistency.

The paper proposes G-buffers as an intermediate representation for generative rendering. Here, $X$ denotes a collection of intrinsic or rendering-related modalities, including albedo, normals, depth, material parameters, diffuse irradiance, and albedo-free direct illumination. The framework supports both inverse rendering, mapping RGB data to G-buffers ($\mathrm{RGB}\rightarrow X$), and forward rendering, mapping G-buffers to realistic RGB sequences ($X\rightarrow\mathrm{RGB}$). It further extends both directions from still images and fixed-length video clips to streaming generation.

The central methodological claim is that a pretrained video diffusion transformer can be repurposed into a multi-input, multi-output, multimodal renderer without introducing a separate architectural branch for every task. The implementation is based on Wan 2.1 14B, with the DiT parameters finetuned and the VAE frozen. The resulting framework combines frame-wise concatenation, QK type embeddings, clean input tokens, and X-patchify. These mechanisms establish explicit token-level distinctions between modality, input/output role, and diffusion state.

The paper’s contributions are therefore broader than a single forward-rendering model. It presents a general finetuning recipe, a procedure for constructing real-video/G-buffer training pairs, controls that trade explicit scene adherence against generative completion, lighting-specific conditioning signals, and a streaming training strategy combining teacher forcing, self forcing, and long-context tokens.

Repurposing a video DiT for inverse and forward rendering

The base DiT is originally trained to denoise a sequence of noisy RGB latent frames. RGBX-Next reuses the model’s latent-frame interface by assigning some frame slots to clean conditioning signals and others to noisy outputs. In $\mathrm{RGB}\rightarrow X$, clean RGB latents occupy input slots while G-buffer latents are denoised in output slots. In $X\rightarrow\mathrm{RGB}$, the roles are reversed. The flow-matching loss is evaluated only on output modalities; clean inputs are supplied as context and receive no diffusion noise or loss.

This frame-wise organization is preferable to simply concatenating input and output channels. Channel-wise concatenation reduces the number of tokens, but it forces the pretrained model to reinterpret attention patterns developed for within-video temporal interactions as cross-modal conditioning. The authors report substantially worse convergence for this baseline. Frame-wise concatenation instead preserves separate token streams while allowing self-attention to mediate interactions among modalities.

Two modifications make the role distinction explicit. First, QK type embeddings add learned offsets to attention queries and keys based on a token’s role and modality. A clean RGB input token, for example, receives a different type from a noisy albedo output token. The type information is injected directly into QK representations rather than added only to the token embedding, allowing attention compatibility to depend explicitly on modality and denoising role.

Second, clean input tokens receive an effective timestep of zero, while output tokens retain the current diffusion timestep. This avoids applying the same noise-state embedding to clean and noisy tokens. The ablations indicate that both mechanisms matter: QK type embeddings improve estimates across modalities, and clean input tokens provide a further improvement in quality and finetuning convergence.

The approach also supports multiple input modalities. X-patchify concatenates clean G-buffer latents along the channel dimension and maps the resulting multimodal patch into one conditioning token. Output modalities remain separately tokenized. This distinction is important: the authors report that packing multiple noisy output modalities into a single token stream causes convergence problems, whereas packing clean inputs preserves quality while reducing conditioning-token overhead.

The paper further demonstrates a proof-of-concept unified RGB$\times X$ model. A single set of weights can perform inverse rendering, forward rendering, and configurations with multiple inputs or outputs, with token embeddings specifying the selected task. This result supports the paper’s architectural generality claim, although the unified model is not explored as comprehensively as the specialized models.

RGB-to-X inverse rendering

The inverse-rendering component is trained initially on paired synthetic RGB images and G-buffers. The internal synthetic collection contains 900 path-traced video sequences, each with approximately 100 frames, spanning indoor and outdoor scenes. The model predicts albedo, normals, depth, material properties, diffuse irradiance, and direct lighting from RGB input.

A significant result is the estimation of diffuse irradiance from real video. The authors state that, to their knowledge, temporally stable irradiance decomposition from real videos had not previously been demonstrated in this framework. This matters because irradiance is a substantially more useful lighting representation than raw image intensity for downstream forward rendering: it separates diffuse illumination from albedo and can subsequently be used as an explicit control signal.

On the Hypersim test set, RGBX-Next achieves the following results:

Modality	PSNR	SSIM	LPIPS
Albedo	20.17	0.8219	0.1420
Normal	21.22	0.7638	0.1986
Irradiance	25.19	0.8589	0.3535
Depth	29.47	0.9032	0.1166

The model is best on all three reported metrics for albedo and depth, and on two of three metrics for normals and irradiance, relative to RGB↔X [2024] and DiffusionRenderer [2025]. For example, its depth performance reaches 29.47 PSNR and 0.9032 SSIM, compared with 17.72 and 0.6570 for DiffusionRenderer. Its albedo result reaches 20.17 PSNR, 0.8219 SSIM, and 0.1420 LPIPS, improving over RGB↔X on PSNR and SSIM and over both baselines on LPIPS.

These comparisons require qualification. RGBX-Next was not trained on Hypersim, whereas RGB↔X was, so the benchmark is not a strictly matched training-distribution comparison. Conversely, Hypersim is synthetic, and the paper correctly notes that the quality of real-video decomposition is better assessed visually. Material properties are not quantitatively evaluated because reliable ground truth for roughness and metallicity is unavailable.

Qualitatively, the model produces flatter albedo estimates with less residual shading than RGB↔X and DiffusionRenderer. It also avoids a sky-depth failure observed in DiffusionRenderer. Normals are visually comparable to DiffusionRenderer, while irradiance prediction provides functionality absent from the cited baselines. The use of a video DiT and temporally coupled inference yields coherent estimates over 17-frame clips and, after streaming finetuning, over substantially longer sequences.

Realistic forward rendering from estimated G-buffers

The forward-rendering problem is more difficult than inverse decomposition because the model must produce realistic RGB appearance from representations that may be incomplete, simplified, or imperfectly estimated. Training solely on synthetic RGB/G-buffer pairs causes the output distribution to acquire a synthetic appearance. The paper presents this as a strong empirical claim: synthetic paired data is insufficient for realistic $X\rightarrow\mathrm{RGB}$ rendering, even when the conditioning G-buffers are geometrically valid.

To address this issue, the authors construct a real-video dataset containing approximately 6,622 Pexels videos, with 371 held out for testing. Each clip is annotated with G-buffers estimated by the RGB-to-X model. VLM-generated captions identify whether content appears photographic, real, rendered, or synthetic. These captions enable classifier-free guidance toward realistic appearance while retaining the pretrained text-conditioning interface.

This data construction introduces a form of self-training or pseudo-labeling: the G-buffers used to train the forward renderer are predictions of the inverse renderer rather than ground truth. Its advantage is distributional alignment between training and deployment, because the forward model learns to render from the same imperfect estimates it will receive at inference. Its limitation is that inverse-rendering errors can become part of the conditioning distribution and may constrain what the forward model learns.

The $X\rightarrow\mathrm{RGB}$ models support specialized conditioning configurations, including albedo, normals, depth, albedo with irradiance, and albedo with direct lighting. They also support a generalized model trained with modality dropout over albedo, normal, irradiance, and material inputs. The resulting outputs follow the supplied G-buffers while synthesizing omitted information such as texture detail, camera appearance, local lighting, and material response.

On the held-out real-video test set, the albedo-conditioned model obtains an FID of 45.3871. This compares with 62.2883 for VACE, 62.3884 for RGB↔X, and 88.6039 for the channel-wise concatenation baseline. The result supports two distinct conclusions. First, training on real videos paired with estimated G-buffers substantially improves distributional realism. Second, the proposed token organization provides better conditioning than both channel-wise concatenation and VACE under matched training conditions.

The comparison with RGB↔X is not entirely architectural or data matched, since RGB↔X is an image model and was trained on indoor-scene data. Nevertheless, the qualitative evidence is consistent with the FID result: RGB↔X often produces flat or unrealistic outputs and can fail to follow the albedo input, whereas RGBX-Next preserves albedo structure while generating more photographic appearance. VACE and channel-wise concatenation produce broadly realistic images but follow the conditioning less reliably.

Control of realism, appearance, and lighting

RGBX-Next separates scene specification from appearance synthesis through several inference-time controls. The primary realism control uses positive and negative text prompts in CFG. A positive prompt describes photographic, high-resolution, realistic imagery, while a negative prompt describes synthetic, rendered, or game-like appearance. Reversing these prompts shifts the output toward a more synthetic and stylized distribution while retaining the broad geometry and colors implied by the G-buffers.

This result exposes an important property of the framework: G-buffers constrain scene content but do not uniquely determine image appearance. Text guidance can alter lighting, texture statistics, glossiness, and camera-like characteristics without necessarily violating the supplied albedo or geometry. Consequently, the model is not a physically faithful renderer in the conventional sense; it is a conditional generative renderer whose outputs are controlled by both explicit buffers and learned semantic priors.

The authors introduce condition dropout, spatial dropout, and blurring to regulate the strength and granularity of control. A modality can be entirely removed, spatially masked, or blurred. These transformations are applied during training so that zero-valued or blurred regions are interpreted as absent or weak guidance rather than as literal black or blurred scene content. Conditions can also be used only during early denoising iterations, allowing coarse structure or lighting to guide composition while leaving later iterations freer to synthesize details.

Spatial dropout is particularly relevant to practical editing. When albedo is specified only in selected regions, a model trained without segment dropout tends to produce flat content in the unconditioned areas. Training with randomly removed segments allows the model to preserve visual richness where no explicit albedo is provided. The paper uses SAM 2 to obtain video segments for this augmentation.

Two lighting representations are proposed. Diffuse irradiance is physically interpretable and relatively inexpensive to compute compared with full path tracing because it can be estimated through simpler Lambertian transport. The second, albedo-free direct lighting buffer removes diffuse and specular albedo factors from direct illumination. It is cheaper to compute and remains approximately orthogonal to albedo, but it excludes multi-bounce lighting. The experiments show that both can guide appearance, although the buffers are deliberately blurred or removed during later denoising iterations to avoid overconstraining the output.

The prompt-control experiment also illustrates the limits of buffer-only conditioning. An albedo-conditioned rendering misses warm ambient illumination from emissive yellow strip lights because that information is not encoded in the albedo buffer. Adding a scene-specific textual description restores warmer illumination and changes the robot’s appearance from diffuse to glossy. This demonstrates that the generative prior can supply missing scene information, but it also means that the output is not determined solely by the G-buffers.

Streaming rendering and temporal stability

Fixed-length video models cannot directly process unbounded sequences. RGBX-Next therefore uses a $1/16$ streaming formulation in which each chunk contains a stitching frame from the previous chunk and 16 newly conditioned input frames. The model generates the corresponding output frames, and the generated endpoint becomes the stitching frame for the next chunk.

Teacher forcing alone creates a train-test mismatch: training uses ground-truth stitching frames, whereas inference feeds back model predictions. The paper observes blur and error accumulation after approximately 100–200 frames. Brightness perturbations, blur or sharpening, and VAE round-trip corruption of the stitching frame provide only a modest improvement.

Long-context tokens append a clean reference frame to every chunk. This reference supplies sequence-level information about color, lighting, and material appearance and reduces forgetting over long temporal intervals. However, the authors report that long-context conditioning alone does not eliminate drift, which becomes visible on sequences exceeding approximately 600 frames.

The final strategy combines augmented teacher forcing with hybrid teacher/self forcing. During self forcing, the model’s own sampled output is used to construct the noisy training latent for subsequent chunks. This exposes training to the prediction errors that occur at inference. Self forcing alone is unstable and can replace drift with inter-chunk flicker, so the authors first train a teacher-forced model and then continue with an interleaved teacher/self-forcing stage at a learning rate ten times smaller.

The resulting streaming RGB-to-X model is reported to remain stable without catastrophic drift for approximately 1,000 frames. Streaming X-to-RGB results maintain temporal coherence and lighting consistency on both synthetic and estimated albedo sequences. These findings support the specific training prescription, although the evidence is primarily qualitative and presented through long supplementary videos rather than a comprehensive temporal metric suite.

The models are not yet interactive in the reported configuration. Inference uses 20 diffusion steps, and a 17-frame X-to-RGB segment takes approximately 400 seconds on a single A100 80GB GPU. Streaming inference scales approximately with the number of chunks. Thus, the paper demonstrates long-context stability, not real-time deployment. It explicitly leaves distillation using methods such as DMD and DMD2 unapplied.

Limitations and open questions

The paper acknowledges that the optimality of G-buffers as a generative-rendering representation remains unestablished. The experiments show that albedo, geometry, and lighting buffers provide useful control, but they do not establish that G-buffers are superior to alternative representations such as neural scene descriptors, learned 3D context, image-space illumination fields, or richer scene graphs.

The training pipeline also relies on estimated G-buffers for real videos. This is necessary to avoid the synthetic appearance induced by synthetic-only forward-rendering data, but it creates dependence between the inverse and forward models. The paper does not provide a systematic analysis of how inverse-rendering errors propagate into forward-rendering fidelity, nor does it quantify the degree to which generated outputs remain consistent with physically valid transport.

The evaluation is likewise incomplete in several respects. Quantitative inverse-rendering results are reported on Hypersim despite a training-distribution mismatch, while forward-rendering evaluation relies mainly on FID and qualitative inspection. There is no reported metric for temporal identity preservation, geometric consistency, buffer-to-image adherence, or lighting-control accuracy. Material properties are not quantitatively evaluated because suitable ground truth is unavailable. These omissions leave open whether improvements in perceptual realism are accompanied by improved physical consistency.

The streaming design uses a single reference frame as long-context memory. This reduces drift but may not preserve object identity, scene topology, or appearance changes that cannot be represented by one frame. The authors identify extension to multiple reference frames or a true 3D context as an open question. Similarly, spatial dropout currently distinguishes between specified and unspecified regions, but does not provide a continuous object- or material-level conditioning representation.

Finally, the models retain the computational cost of the 14B Wan 2.1 teacher. The paper’s strongest deployment claim is therefore conditional: the framework could become practical after distillation, but no distilled model or measured interactive system is presented. The question left open is whether distillation can preserve the demonstrated G-buffer adherence, realism control, and long-horizon temporal stability simultaneously.

Conclusion

RGBX-Next presents a coherent DiT-based framework for bidirectional generative rendering. Its principal technical contribution is a token-level conditioning design that distinguishes modality, input/output role, and noise state while supporting multiple modalities within a common video-transformer architecture. The empirical results show strong inverse-rendering performance, realistic forward rendering from both synthetic and estimated G-buffers, explicit lighting control, and streaming stability over approximately 1,000 frames.

The most consequential empirical findings are that real-video supervision with estimated G-buffers is important for avoiding synthetic appearance, and that hybrid self-forcing with long-context tokens is necessary for stable long-video rendering. The framework nevertheless remains a computationally expensive teacher system, and its physical consistency, representation optimality, and quantitative long-term behavior require further evaluation.