---
title: G-Buffer-Guided Video Generation
url: https://www.emergentmind.com/topics/g-buffer-guided-video-generation
type: topic
---

# G-Buffer-Guided Video Generation

G-buffer-guided video generation denotes a class of methods in which video synthesis or editing is controlled by explicit scene buffers derived from graphics or reconstruction pipelines rather than by text alone. In the most literal sense, the guidance signal is a physically oriented per-pixel representation of scene geometry and materials—typically normals, depth, albedo, roughness, and metallic—which is consumed by a generative model to produce photorealistic frames or videos. In a broader but closely related sense, the field also includes methods that replace a classical G-buffer with rendered geometry proxies, Gaussian primitive buffers, or render-derived confidence and inconsistency maps, while preserving the same underlying objective: constrain generation with structured scene information so that geometry, appearance, lighting, and temporal behavior remain controllable and coherent across frames. Representative systems include unified inverse/forward neural renderers such as DiffusionRenderer [2501.18590], geometry-informed refiners for 3D Gaussian Splatting such as GaussFusion [2603.25053], autonomous-driving weather editors such as AutoWeather4D [2603.26546], and interactive frame renderers such as FrameDiffuser [2512.16670].

## 1. Definition and scope

In the narrow graphics sense, a G-buffer is a rasterized set of per-pixel scene attributes that separates geometry and material state from final image formation. DiffusionRenderer provides a canonical example of this formulation for video generation, using surface normals \(n \in \mathbb{R}^{F \times H \times W \times 3}\), relative depth \(d \in \mathbb{R}^{F \times H \times W \times 1}\), base color or albedo \(a \in \mathbb{R}^{F \times H \times W \times 3}\), roughness \(r \in \mathbb{R}^{F \times H \times W \times 1}\), and metallic \(m \in \mathbb{R}^{F \times H \times W \times 1}\) [2501.18590]. AutoWeather4D adopts a related intrinsic buffer set for dynamic driving scenes, explicitly using metric depth \(\mathbf{D}\), surface normals \(\mathbf{N}\), albedo or base color \(\mathbf{A}\), metallic \(\mathbf{M}\), and roughness \(\mathbf{R}\), with semantic masks as auxiliary structural priors [2603.26546]. FrameDiffuser uses a 10-channel ControlNet input composed of 9 G-buffer channels plus 1 irradiance channel: basecolor, normals, depth, roughness, metallic, and predicted irradiance [2512.16670].

A broader interpretation includes methods whose guidance is structured and render-derived but not a classical deferred stack. GaussFusion rasterizes a Gaussian Primitives Buffer \(\mathcal{G} = \{\mathbf{C}, A, D, \mathbf{N}, \mathbf{U}\}\), consisting of rendered color, opacity, depth, normals, and an uncertainty map derived from inverse projected covariance [2603.25053]. GS-DiT conditions a Video DiT on an RGB video rendered from a pseudo 4D Gaussian field rather than on explicit depth or normal channels, and is therefore “Yes, in spirit” but “No, in the classical graphics sense” as a G-buffer-guided method [2501.02690]. Video4DGen uses rendered Dynamic Gaussian Surfels output \(r^m\) plus a confidence mask \(\mathcal{M}^m\) rather than a standard stacked G-buffer [2504.04153]. I2V3D likewise uses coarse rendered RGB, depth maps, and inversion-derived features rather than normals, albedo, or material IDs as separate conditioning channels [2503.09733].

A persistent misconception is that any geometry-guided video model is automatically G-buffer-guided in the strict sense. The literature instead shows a spectrum. At one end are explicitly intrinsic, PBR-oriented interfaces such as DiffusionRenderer and AutoWeather4D. At the other are rendered-scene-conditioning approaches such as GS-DiT, Video4DGen, I2V3D, and G\(^2\)Editor, where the guidance is structured and rasterized but not fully factorized into standard graphics buffers [2508.20471].

## 2. Representational forms and conditioning interfaces

The defining technical question is not only which structured signals are used, but how they are injected into the generator. DiffusionRenderer separates conditioning paths according to spatial alignment. Pixel-aligned G-buffer latents are concatenated with the noisy image latent at the diffusion UNet input, while non-aligned environment lighting is encoded through a dedicated environment-map encoder and injected via repurposed cross-attention layers [2501.18590]. The latent G-buffer tensor is
\[
g = \{E(n), E(d), E(a), E(r), E(m)\} \in \mathbb{R}^{F \times h \times w \times 20},
\]
and the conditional denoiser is written as
\[
f_\theta(z_t; g, c_{\text{env}}, t).
\]
The method does not explicitly evaluate the rendering equation
\[
L_o(p, \omega_o) = \int_{\Omega} f_r(p, \omega_o, \omega_i) L_i(p, \omega_i) \left| n \cdot \omega_i \right| d\omega_i,
\]
but instead learns a mapping from geometry, material, and lighting conditions to appearance [2501.18590].

GaussFusion departs from input concatenation and instead introduces a Geometry Adapter. Each GP-Buffer modality is VAE-encoded separately, concatenated into a unified geometry latent, projected by a 3D convolution, processed by self-attention, and then injected into the main Wan DiT backbone through interleaved adapter blocks [2603.25053]. The adapter is designed as a parallel side-network that injects hierarchical geometric features into corresponding DiT layers rather than simply adding conditioning latents to noisy video latents. The empirical rationale is explicit: replacing the Geometry Adapter with simple convolutional fusion yields lower PSNR, SSIM, LPIPS, and FID than the full adapter [2603.25053].

FrameDiffuser also adopts dual conditioning, but in an autoregressive frame renderer rather than a sequence model. Structural guidance from current G-buffer plus irradiance is handled by ControlNet, while temporal coherence from the previous frame is handled by ControlLoRA [2512.16670]. The irradiance channel is computed as
\[
I_{t-1}(x,y) = \frac{L(F_{t-1}(x,y))}{L(C_{t-1}(x,y)) + \epsilon},
\]
with values clamped to \([0,2]\) and normalized for network input. This design assigns current-frame structure to ControlNet and temporal continuity to latent previous-frame conditioning modulated by low-rank adaptation [2512.16670].

Driving-video object editing and image-to-video generation expose additional interface patterns. G\(^2\)Editor concatenates VAE features of the masked video, Gaussian video renderings, and edge masks into the inpainting UNet input, while depth-aware scene boxes are encoded into multi-scale features and fused into UNet blocks [2508.20471]. I2V3D combines depth ControlNet with rendered-feature control derived from DDIM or EDM inversion, using self-attention and convolutional features extracted from coarse rendered frames to preserve layout and fine geometric structure [2503.09733].

| Method | Guidance representation | Injection pattern |
|---|---|---|
| DiffusionRenderer | Normals, depth, albedo, roughness, metallic, environment map | Latent concatenation for pixel-aligned buffers; cross-attention for lighting [2501.18590] |
| GaussFusion | GP-Buffer: color, alpha, depth, normals, uncertainty | Interleaved Geometry Adapter blocks in Wan DiT [2603.25053] |
| FrameDiffuser | Basecolor, normals, depth, roughness, metallic, irradiance | ControlNet for structure; ControlLoRA for temporal coherence [2512.16670] |
| G\(^2\)Editor | Gaussian video, depth-aware boxes, edge masks, edit mask | Input concatenation plus multiscale feature fusion [2508.20471] |

This suggests a general design principle: spatially aligned scene state is commonly handled by concatenation or residual control pathways, whereas non-aligned, global, or semantically different signals are more often routed through attention or dedicated adapters.

## 3. Inverse rendering, forward rendering, and render-derived guidance

One major branch of the field treats G-buffer-guided video generation as neural forward rendering conditioned on inferred scene attributes. DiffusionRenderer makes the inverse-to-forward linkage explicit. The inverse renderer predicts one attribute at a time among \(\{n,d,a,r,m\}\) from RGB video, using RGB latent conditioning by concatenation and attribute identity through cross-attention. The target attribute latent follows
\[
g_t^P = \alpha_t g_0^P + \sigma_t \epsilon,
\]
and the denoiser is
\[
f_\theta(g_t^P; z, c_{\text{emb}}^P, t),
\]
with the loss
\[
\mathcal{L}(\theta, c_{\text{emb}}) = \left\| f_\theta \left(g_t^P; z, c_{\text{emb}}^P, t \right) - g_0^P \right\|_2^2.
\]
At inference time, the workflow is direct:
\[
\{ \hat{n}, \hat{d}, \hat{a}, \hat{r}, \hat{m} \} = \text{InverseRenderer}(I),
\]
followed by
\[
\hat{I}_{\text{tgt}} = \text{ForwardRenderer}(\{\hat{n}, \hat{d}, \hat{a}, \hat{r}, \hat{m}, E_{\text{tgt}}\}).
\]
This makes relighting, material editing, and object insertion possible from a single RGB video [2501.18590].

A second branch uses render-derived guidance to repair or refine an existing 3D representation. GaussFusion begins from an already reconstructed 3DGS scene, rasterizes a temporally ordered GP-Buffer video, and uses a latent flow-matching video generator to refine the renderings [2603.25053]. The same outputs can then be reused as pseudo-observations to update the underlying 3DGS with the standard photometric 3DGS loss. GaussVideoDreamer operates on a related alternating loop for static 3D scene generation: infer coarse geometry from a single image, render novel views and masks, inpaint progressively, fit 3D Gaussian splats, detect inconsistent rendered regions via residual-derived masks, and feed rendered views plus change-map and depth guidance into video diffusion before optimizing 3DGS again [2504.10001].

A third branch uses rendered structured scenes as an implicit substitute for a classical G-buffer. GS-DiT constructs a pseudo 4D Gaussian field from dense 3D point tracking, renders videos from that field, and conditions CogVideoX by concatenating rendered guidance latents with noisy latents [2501.02690]. The rendered RGB guidance encodes 3D scene layout, motion over time, visibility, camera viewpoint, camera intrinsics effects, and potentially edited object motion, even though depth and normals are not fed as separate channels [2501.02690]. Video4DGen similarly reconstructs a Dynamic Gaussian Surfels representation and then guides denoising by rendering the reconstructed dynamic subject from target viewpoints, using a confidence map derived from normal agreement to decide where geometry-backed guidance should dominate [2504.04153].

A plausible implication is that G-buffer-guided generation now spans two distinct but converging paradigms: explicit intrinsic control, where generation approximates a rendering equation over compact scene buffers, and rendered-proxy control, where the generator consumes a rasterized scene model that already encodes geometry, visibility, and pose but not necessarily a factorized material decomposition.

## 4. Temporal coherence and video-native modeling

Temporal stability is a central differentiator between image-conditioned rendering and video-native generation. DiffusionRenderer is explicitly not an image model applied frame by frame with an external consistency loss; both inverse and forward rendering are built on Stable Video Diffusion, and the paper reports better perceptual temporal quality using ColorVideoVDP [2501.18590]. Temporal consistency is handled implicitly through joint latent denoising over the full clip rather than through custom optical flow losses, recurrent propagation, or explicit temporal attention modifications. On relighting, CVVDP scores are 6.77 on SyntheticObjects and 6.40 on SyntheticScenes for DiffusionRenderer, versus 6.49 and 3.47 for Neural Gaffer and 5.44 and 2.99 for DiLightNet [2501.18590].

GaussFusion also omits explicit optical flow modules, recurrent state, and separate temporal-coherence losses. Its temporal coherence arises from a native video latent model with spatiotemporal VAE latents and transformer processing over the entire clip [2603.25053]. The context window is 81 frames, and for longer videos the method uses a bidirectional sliding-window strategy in which the last frame of the current prediction becomes the first frame of the next window; the paper notes that this supports short-term continuity but does not fully solve long-term coherence [2603.25053].

FrameDiffuser addresses a different temporal problem: causal long-horizon consistency when future frames are unavailable. After initialization from a starting frame and corresponding irradiance, it generates one frame at a time autoregressively, using current G-buffer and its own previous output [2512.16670]. The method identifies a train-test mismatch problem specific to autoregressive rendering and addresses it with a three-stage training strategy culminating in self-conditioning. During stage 3, 50% of the frames are generated frames, and the previous-frame conditioning is corrupted as
\[
\mathbf{x}'_{t-\delta} = \mathbf{x}_{t-\delta} + \sigma \cdot \boldsymbol{\epsilon}, \quad \sigma \sim \mathcal{U}(0.0, 0.2), \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, \mathbf{I}),
\]
with temporal offsets \(\delta \in \{-2,\ldots,2\}\) and higher weights for adjacent frames [2512.16670]. On a 589-frame Electric Dreams sequence, adding self-conditioning and noise injection improves PSNR from 12.29 to 18.16, SSIM from 0.323 to 0.454, and LPIPS from 0.431 to 0.253 [2512.16670].

Other systems use different temporal mechanisms. G\(^2\)Editor treats object editing as video inpainting and adds temporal-attention layers after cross-attention layers in the inpainting UNet, training only these temporal layers in a second stage [2508.20471]. I2V3D uses a keyframe-plus-interpolation decomposition: jointly generate sparse keyframes with Extended Attention across frames, then perform training-free bidirectional interpolation with depth ControlNet and rendered feature overrides [2503.09733]. AutoWeather4D derives temporal consistency primarily from 3D anchoring rather than recurrent generative modeling: Pi3 provides spatiotemporally coherent 4D depth, puddles are evaluated in world coordinates, local lights are reconstructed in 3D, particles follow explicit kinematics, and the rendered sequence serves as a temporally coherent conditioning source for VidRefiner [2603.26546].

A common misconception is that temporal coherence in this literature is mainly enforced by optical flow or explicit warping. Several representative systems instead rely on native video diffusion backbones, autoregressive self-conditioning, or physically anchored world-space edits rather than on hand-crafted motion losses.

## 5. Data regimes, supervision, and synthetic–real transfer

Because dense supervision over geometry, material, lighting, and video appearance is rare in real footage, many methods depend on hybrid supervision strategies. DiffusionRenderer generates a synthetic dataset with a custom OptiX path tracer containing \(\{I, a, r, m, n, d, E\}\), with 150,000 videos at 24 frames and \(512 \times 512\) resolution [2501.18590]. To reduce the synthetic-to-real gap, its inverse renderer—trained on synthetic videos plus InteriorVerse and HyperSim—is applied to real videos from DL3DV10k to generate pseudo-G-buffers, while environment maps are estimated with DiffusionLight. Each real video is split into 15 segments, yielding around 150,000 auto-labeled real video samples, and the forward renderer is then jointly trained on synthetic data and pseudo-labeled real data, with a LoRA branch \(\Delta\) for the real-data term [2501.18590].

GaussFusion similarly centers its robustness strategy on paired artifact synthesis rather than on purely photometric self-supervision. It renders paired videos from high-quality and deliberately degraded 3DGS reconstructions, including sparse-view simulation by retaining only 5% of original video frames, multiple initialization strategies, underfitting, and degradation from feed-forward predictors such as DepthSplat and MVSplat [2603.25053]. The curated training set contains 75K+ paired video samples, each with 81 frames, from DL3DV and RE10K [2603.25053].

GS-DiT avoids multi-view video supervision by constructing a pseudo 4D Gaussian field directly from monocular videos using dense 3D point tracking. Its D3D-PT tracker is trained on 11,000 24-frame RGB-D sequences generated in Kubric at \(512 \times 512\) resolution for 500,000 iterations on 8× NVIDIA A100 80GB, while GS-DiT itself is trained on 400K WebVid-10M clips for 100,000 steps at \(320 \times 512\) [2501.02690]. AutoWeather4D is notable for relying heavily on pretrained components and analytical rendering rather than on bespoke paired weather-conversion supervision: Pi3 for 4D reconstruction, DiffusionRenderer for inverse and forward rendering, OWL-ViT and SAM2 for semantics, Grounded-SAM for sky masking, WAN-FUN for refinement, and sparse LiDAR or a camera-height prior for depth calibration [2603.26546].

FrameDiffuser adopts a different deployment assumption: environment-specific specialization. It trains separate models for six Unreal Engine 5 environments—Electric Dreams, City Sample, Hillside Sample Project, Downtown West, City Park, and Derelict Corridor—using consecutive frame pairs with corresponding G-buffers [2512.16670]. The method explicitly prioritizes environment-specific consistency and inference speed over broad generalization.

These strategies indicate that G-buffer-guided video generation is not tied to a single supervision regime. Fully supervised synthetic rendering, pseudo-label bootstrapping, realistic artifact simulation, per-domain specialization, and analytical scene editing all coexist, with the chosen regime reflecting whether the task is relighting, refinement, weather conversion, object editing, or interactive rendering.

## 6. Applications, empirical behavior, and limitations

The application space is unusually broad because explicit scene buffers expose semantically meaningful controls. DiffusionRenderer demonstrates relighting, material editing, and realistic object insertion from a single video input [2501.18590]. On forward rendering with ground-truth G-buffers and lighting, it achieves 26.0 / 0.780 / 0.201 PSNR/SSIM/LPIPS on SyntheticScenes, compared to 18.5 / 0.645 / 0.302 for RGB\(\leftrightarrow\)X and 20.7 / 0.630 / 0.300 for DiLightNet, and reaches 28.3 / 0.935 / 0.048 on SyntheticObjects [2501.18590]. On relighting, it reports 27.50 / 0.918 / 0.067 on SyntheticObjects and 24.63 / 0.756 / 0.257 on SyntheticScenes [2501.18590].

GaussFusion targets reconstruction repair rather than relighting or editing. On DL3DV, its full model reaches PSNR 22.548, SSIM 0.832, LPIPS 0.278, and FID 3.933; on RE10K it reaches 28.652, 0.944, 0.180, and 6.672 [2603.25053]. Its modality ablation is especially informative for the broader topic: RGB-only conditioning gives 19.148 PSNR, 0.719 SSIM, 0.385 LPIPS, and 15.451 FID, while the full five-modality GP-Buffer gives 20.753, 0.773, 0.329, and 6.724, showing that every additional channel helps [2603.25053].

AutoWeather4D shows how explicit intrinsic buffers can support physically controlled weather conversion. Evaluated on 120 Waymo Open Dataset scenes and 27,360 frames, it reports CLIP score 0.2586, Vehicle 3D detection IoU 0.915, Vehicle CLIP cosine similarity 0.871, and Human evaluation 0.826, with depth si-RMSE 0.247, Edge F1 0.129, FVD 886.8, and DOVER 0.370 [2603.26546]. The paper emphasizes correct physical edits—removing inherited sunny shadows, producing localized headlight and streetlight effects, synthesizing snow accumulation and road ripples, and preserving dynamic vehicle structure—over pure perceptual scores [2603.26546].

FrameDiffuser addresses interactive neural forward rendering. Averaged across validation sets from its training distributions, it improves over X\(\rightarrow\)RGB from RGB\(\leftrightarrow\)X from 8.60 / 0.3566 / 0.5150 to 18.34 / 0.6377 / 0.2129 in PSNR/SSIM/LPIPS [2512.16670]. Its supplementary 24-frame comparison against DiffusionRenderer reports 20.96 / 0.6378 / 0.2030, compared with 13.03 / 0.4683 / 0.4583 [2512.16670]. The method can roll out over hundreds to thousands of frames, but it remains at approximately 1 frame per second on an RTX 4090 with 10 denoising steps and DPMSolver, so it is not yet real-time [2512.16670].

Several limitations recur across the literature. DiffusionRenderer remains tied to the quality of predicted inverse-rendering buffers and estimated real-world lighting, can introduce slight variations in color or texture during editing, and is relatively expensive and offline rather than interactive [2501.18590]. GaussFusion depends on having a usable initial 3DGS, degrades on rapid motion or severe motion blur because the VAE struggles to encode clean latents, and does not guarantee long-term coherence beyond its windowed strategy [2603.25053]. AutoWeather4D notes failure cases involving extreme long-tail dynamics such as vehicle splash, sensitivity to external feed-forward estimators, and the lack of an emissive channel, which causes self-illuminating objects like traffic lights to darken incorrectly when global illumination is reduced [2603.26546]. FrameDiffuser sacrifices broad generalization through environment specialization, shows out-of-distribution style drift, inherits temporal flicker from the Stable Diffusion 1.5 VAE, and has no explicit long-horizon reset mechanism [2512.16670].

Taken together, these systems indicate that G-buffer-guided video generation is best understood not as a single architecture, but as a family of structured-conditioning strategies for controllable video synthesis. The common thread is the use of rasterized or render-derived scene state to restrict the generative search space. The main axes of variation are the exact buffer representation, the conditioning pathway, whether generation is offline or causal, and whether the model learns a neural approximation to rendering, a refinement operator over imperfect renders, or a physically grounded editor whose diffusion stage is deliberately constrained.

Source: https://www.emergentmind.com/topics/g-buffer-guided-video-generation