G-Buffer-Guided Video Generation

• G-buffer-guided video generation is defined by using explicit per-pixel attributes (normals, depth, albedo, roughness, metallic) as controllable guidance for video synthesis.
• Techniques employ adaptive feature fusion methods such as cross-attention and dedicated geometry adapters to integrate structured, spatially aligned scene information into the generative process.
• Applications range from relighting and weather editing to interactive frame rendering, with strategies ensuring temporal coherence through autoregressive and windowed modeling approaches.

G-buffer-guided video generation denotes a class of methods in which video synthesis or editing is controlled by explicit scene buffers derived from graphics or reconstruction pipelines rather than by text alone. In the most literal sense, the guidance signal is a physically oriented per-pixel representation of scene geometry and materials—typically normals, depth, albedo, roughness, and metallic—which is consumed by a generative model to produce photorealistic frames or videos. In a broader but closely related sense, the field also includes methods that replace a classical G-buffer with rendered geometry proxies, Gaussian primitive buffers, or render-derived confidence and inconsistency maps, while preserving the same underlying objective: constrain generation with structured scene information so that geometry, appearance, lighting, and temporal behavior remain controllable and coherent across frames. Representative systems include unified inverse/forward neural renderers such as DiffusionRenderer (Liang et al., 30 Jan 2025), geometry-informed refiners for 3D Gaussian Splatting such as GaussFusion (Zhu et al., 26 Mar 2026), autonomous-driving weather editors such as AutoWeather4D (Liu et al., 27 Mar 2026), and interactive frame renderers such as FrameDiffuser (Beisswenger et al., 18 Dec 2025).

1. Definition and scope

In the narrow graphics sense, a G-buffer is a rasterized set of per-pixel scene attributes that separates geometry and material state from final image formation. DiffusionRenderer provides a canonical example of this formulation for video generation, using surface normals $n \in \mathbb{R}^{F \times H \times W \times 3}$, relative depth $d \in \mathbb{R}^{F \times H \times W \times 1}$, base color or albedo $a \in \mathbb{R}^{F \times H \times W \times 3}$, roughness $r \in \mathbb{R}^{F \times H \times W \times 1}$, and metallic $m \in \mathbb{R}^{F \times H \times W \times 1}$ (Liang et al., 30 Jan 2025). AutoWeather4D adopts a related intrinsic buffer set for dynamic driving scenes, explicitly using metric depth $\mathbf{D}$, surface normals $\mathbf{N}$, albedo or base color $\mathbf{A}$, metallic $\mathbf{M}$, and roughness $\mathbf{R}$, with semantic masks as auxiliary structural priors (Liu et al., 27 Mar 2026). FrameDiffuser uses a 10-channel ControlNet input composed of 9 G-buffer channels plus 1 irradiance channel: basecolor, normals, depth, roughness, metallic, and predicted irradiance (Beisswenger et al., 18 Dec 2025).

A broader interpretation includes methods whose guidance is structured and render-derived but not a classical deferred stack. GaussFusion rasterizes a Gaussian Primitives Buffer $d \in \mathbb{R}^{F \times H \times W \times 1}$0, consisting of rendered color, opacity, depth, normals, and an uncertainty map derived from inverse projected covariance (Zhu et al., 26 Mar 2026). GS-DiT conditions a Video DiT on an RGB video rendered from a pseudo 4D Gaussian field rather than on explicit depth or normal channels, and is therefore “Yes, in spirit” but “No, in the classical graphics sense” as a G-buffer-guided method (Bian et al., 5 Jan 2025). Video4DGen uses rendered Dynamic Gaussian Surfels output $d \in \mathbb{R}^{F \times H \times W \times 1}$1 plus a confidence mask $d \in \mathbb{R}^{F \times H \times W \times 1}$2 rather than a standard stacked G-buffer (Wang et al., 5 Apr 2025). I2V3D likewise uses coarse rendered RGB, depth maps, and inversion-derived features rather than normals, albedo, or material IDs as separate conditioning channels (Zhang et al., 12 Mar 2025).

A persistent misconception is that any geometry-guided video model is automatically G-buffer-guided in the strict sense. The literature instead shows a spectrum. At one end are explicitly intrinsic, PBR-oriented interfaces such as DiffusionRenderer and AutoWeather4D. At the other are rendered-scene-conditioning approaches such as GS-DiT, Video4DGen, I2V3D, and G$d \in \mathbb{R}^{F \times H \times W \times 1}$3Editor, where the guidance is structured and rasterized but not fully factorized into standard graphics buffers (Li et al., 28 Aug 2025).

2. Representational forms and conditioning interfaces

The defining technical question is not only which structured signals are used, but how they are injected into the generator. DiffusionRenderer separates conditioning paths according to spatial alignment. Pixel-aligned G-buffer latents are concatenated with the noisy image latent at the diffusion UNet input, while non-aligned environment lighting is encoded through a dedicated environment-map encoder and injected via repurposed cross-attention layers (Liang et al., 30 Jan 2025). The latent G-buffer tensor is

$d \in \mathbb{R}^{F \times H \times W \times 1}$4

and the conditional denoiser is written as

$d \in \mathbb{R}^{F \times H \times W \times 1}$5

The method does not explicitly evaluate the rendering equation

$d \in \mathbb{R}^{F \times H \times W \times 1}$6

but instead learns a mapping from geometry, material, and lighting conditions to appearance (Liang et al., 30 Jan 2025).

GaussFusion departs from input concatenation and instead introduces a Geometry Adapter. Each GP-Buffer modality is VAE-encoded separately, concatenated into a unified geometry latent, projected by a 3D convolution, processed by self-attention, and then injected into the main Wan DiT backbone through interleaved adapter blocks (Zhu et al., 26 Mar 2026). The adapter is designed as a parallel side-network that injects hierarchical geometric features into corresponding DiT layers rather than simply adding conditioning latents to noisy video latents. The empirical rationale is explicit: replacing the Geometry Adapter with simple convolutional fusion yields lower PSNR, SSIM, LPIPS, and FID than the full adapter (Zhu et al., 26 Mar 2026).

FrameDiffuser also adopts dual conditioning, but in an autoregressive frame renderer rather than a sequence model. Structural guidance from current G-buffer plus irradiance is handled by ControlNet, while temporal coherence from the previous frame is handled by ControlLoRA (Beisswenger et al., 18 Dec 2025). The irradiance channel is computed as

$d \in \mathbb{R}^{F \times H \times W \times 1}$7

with values clamped to $d \in \mathbb{R}^{F \times H \times W \times 1}$8 and normalized for network input. This design assigns current-frame structure to ControlNet and temporal continuity to latent previous-frame conditioning modulated by low-rank adaptation (Beisswenger et al., 18 Dec 2025).

Driving-video object editing and image-to-video generation expose additional interface patterns. G$d \in \mathbb{R}^{F \times H \times W \times 1}$9Editor concatenates VAE features of the masked video, Gaussian video renderings, and edge masks into the inpainting UNet input, while depth-aware scene boxes are encoded into multi-scale features and fused into UNet blocks (Li et al., 28 Aug 2025). I2V3D combines depth ControlNet with rendered-feature control derived from DDIM or EDM inversion, using self-attention and convolutional features extracted from coarse rendered frames to preserve layout and fine geometric structure (Zhang et al., 12 Mar 2025).

Method	Guidance representation	Injection pattern
DiffusionRenderer	Normals, depth, albedo, roughness, metallic, environment map	Latent concatenation for pixel-aligned buffers; cross-attention for lighting (Liang et al., 30 Jan 2025)
GaussFusion	GP-Buffer: color, alpha, depth, normals, uncertainty	Interleaved Geometry Adapter blocks in Wan DiT (Zhu et al., 26 Mar 2026)
FrameDiffuser	Basecolor, normals, depth, roughness, metallic, irradiance	ControlNet for structure; ControlLoRA for temporal coherence (Beisswenger et al., 18 Dec 2025)
G$a \in \mathbb{R}^{F \times H \times W \times 3}$0Editor	Gaussian video, depth-aware boxes, edge masks, edit mask	Input concatenation plus multiscale feature fusion (Li et al., 28 Aug 2025)

This suggests a general design principle: spatially aligned scene state is commonly handled by concatenation or residual control pathways, whereas non-aligned, global, or semantically different signals are more often routed through attention or dedicated adapters.

3. Inverse rendering, forward rendering, and render-derived guidance

One major branch of the field treats G-buffer-guided video generation as neural forward rendering conditioned on inferred scene attributes. DiffusionRenderer makes the inverse-to-forward linkage explicit. The inverse renderer predicts one attribute at a time among $a \in \mathbb{R}^{F \times H \times W \times 3}$1 from RGB video, using RGB latent conditioning by concatenation and attribute identity through cross-attention. The target attribute latent follows

$a \in \mathbb{R}^{F \times H \times W \times 3}$2

and the denoiser is

$a \in \mathbb{R}^{F \times H \times W \times 3}$3

with the loss

$a \in \mathbb{R}^{F \times H \times W \times 3}$4

At inference time, the workflow is direct: $a \in \mathbb{R}^{F \times H \times W \times 3}$5 followed by

$a \in \mathbb{R}^{F \times H \times W \times 3}$6

This makes relighting, material editing, and object insertion possible from a single RGB video (Liang et al., 30 Jan 2025).

A second branch uses render-derived guidance to repair or refine an existing 3D representation. GaussFusion begins from an already reconstructed 3DGS scene, rasterizes a temporally ordered GP-Buffer video, and uses a latent flow-matching video generator to refine the renderings (Zhu et al., 26 Mar 2026). The same outputs can then be reused as pseudo-observations to update the underlying 3DGS with the standard photometric 3DGS loss. GaussVideoDreamer operates on a related alternating loop for static 3D scene generation: infer coarse geometry from a single image, render novel views and masks, inpaint progressively, fit 3D Gaussian splats, detect inconsistent rendered regions via residual-derived masks, and feed rendered views plus change-map and depth guidance into video diffusion before optimizing 3DGS again (Hao et al., 14 Apr 2025).

A third branch uses rendered structured scenes as an implicit substitute for a classical G-buffer. GS-DiT constructs a pseudo 4D Gaussian field from dense 3D point tracking, renders videos from that field, and conditions CogVideoX by concatenating rendered guidance latents with noisy latents (Bian et al., 5 Jan 2025). The rendered RGB guidance encodes 3D scene layout, motion over time, visibility, camera viewpoint, camera intrinsics effects, and potentially edited object motion, even though depth and normals are not fed as separate channels (Bian et al., 5 Jan 2025). Video4DGen similarly reconstructs a Dynamic Gaussian Surfels representation and then guides denoising by rendering the reconstructed dynamic subject from target viewpoints, using a confidence map derived from normal agreement to decide where geometry-backed guidance should dominate (Wang et al., 5 Apr 2025).

A plausible implication is that G-buffer-guided generation now spans two distinct but converging paradigms: explicit intrinsic control, where generation approximates a rendering equation over compact scene buffers, and rendered-proxy control, where the generator consumes a rasterized scene model that already encodes geometry, visibility, and pose but not necessarily a factorized material decomposition.

4. Temporal coherence and video-native modeling

Temporal stability is a central differentiator between image-conditioned rendering and video-native generation. DiffusionRenderer is explicitly not an image model applied frame by frame with an external consistency loss; both inverse and forward rendering are built on Stable Video Diffusion, and the paper reports better perceptual temporal quality using ColorVideoVDP (Liang et al., 30 Jan 2025). Temporal consistency is handled implicitly through joint latent denoising over the full clip rather than through custom optical flow losses, recurrent propagation, or explicit temporal attention modifications. On relighting, CVVDP scores are 6.77 on SyntheticObjects and 6.40 on SyntheticScenes for DiffusionRenderer, versus 6.49 and 3.47 for Neural Gaffer and 5.44 and 2.99 for DiLightNet (Liang et al., 30 Jan 2025).

GaussFusion also omits explicit optical flow modules, recurrent state, and separate temporal-coherence losses. Its temporal coherence arises from a native video latent model with spatiotemporal VAE latents and transformer processing over the entire clip (Zhu et al., 26 Mar 2026). The context window is 81 frames, and for longer videos the method uses a bidirectional sliding-window strategy in which the last frame of the current prediction becomes the first frame of the next window; the paper notes that this supports short-term continuity but does not fully solve long-term coherence (Zhu et al., 26 Mar 2026).

FrameDiffuser addresses a different temporal problem: causal long-horizon consistency when future frames are unavailable. After initialization from a starting frame and corresponding irradiance, it generates one frame at a time autoregressively, using current G-buffer and its own previous output (Beisswenger et al., 18 Dec 2025). The method identifies a train-test mismatch problem specific to autoregressive rendering and addresses it with a three-stage training strategy culminating in self-conditioning. During stage 3, 50% of the frames are generated frames, and the previous-frame conditioning is corrupted as

$a \in \mathbb{R}^{F \times H \times W \times 3}$7

with temporal offsets $a \in \mathbb{R}^{F \times H \times W \times 3}$8 and higher weights for adjacent frames (Beisswenger et al., 18 Dec 2025). On a 589-frame Electric Dreams sequence, adding self-conditioning and noise injection improves PSNR from 12.29 to 18.16, SSIM from 0.323 to 0.454, and LPIPS from 0.431 to 0.253 (Beisswenger et al., 18 Dec 2025).

Other systems use different temporal mechanisms. G$a \in \mathbb{R}^{F \times H \times W \times 3}$9Editor treats object editing as video inpainting and adds temporal-attention layers after cross-attention layers in the inpainting UNet, training only these temporal layers in a second stage (Li et al., 28 Aug 2025). I2V3D uses a keyframe-plus-interpolation decomposition: jointly generate sparse keyframes with Extended Attention across frames, then perform training-free bidirectional interpolation with depth ControlNet and rendered feature overrides (Zhang et al., 12 Mar 2025). AutoWeather4D derives temporal consistency primarily from 3D anchoring rather than recurrent generative modeling: Pi3 provides spatiotemporally coherent 4D depth, puddles are evaluated in world coordinates, local lights are reconstructed in 3D, particles follow explicit kinematics, and the rendered sequence serves as a temporally coherent conditioning source for VidRefiner (Liu et al., 27 Mar 2026).

A common misconception is that temporal coherence in this literature is mainly enforced by optical flow or explicit warping. Several representative systems instead rely on native video diffusion backbones, autoregressive self-conditioning, or physically anchored world-space edits rather than on hand-crafted motion losses.

5. Data regimes, supervision, and synthetic–real transfer

Because dense supervision over geometry, material, lighting, and video appearance is rare in real footage, many methods depend on hybrid supervision strategies. DiffusionRenderer generates a synthetic dataset with a custom OptiX path tracer containing $r \in \mathbb{R}^{F \times H \times W \times 1}$0, with 150,000 videos at 24 frames and $r \in \mathbb{R}^{F \times H \times W \times 1}$1 resolution (Liang et al., 30 Jan 2025). To reduce the synthetic-to-real gap, its inverse renderer—trained on synthetic videos plus InteriorVerse and HyperSim—is applied to real videos from DL3DV10k to generate pseudo-G-buffers, while environment maps are estimated with DiffusionLight. Each real video is split into 15 segments, yielding around 150,000 auto-labeled real video samples, and the forward renderer is then jointly trained on synthetic data and pseudo-labeled real data, with a LoRA branch $r \in \mathbb{R}^{F \times H \times W \times 1}$2 for the real-data term (Liang et al., 30 Jan 2025).

GaussFusion similarly centers its robustness strategy on paired artifact synthesis rather than on purely photometric self-supervision. It renders paired videos from high-quality and deliberately degraded 3DGS reconstructions, including sparse-view simulation by retaining only 5% of original video frames, multiple initialization strategies, underfitting, and degradation from feed-forward predictors such as DepthSplat and MVSplat (Zhu et al., 26 Mar 2026). The curated training set contains 75K+ paired video samples, each with 81 frames, from DL3DV and RE10K (Zhu et al., 26 Mar 2026).

GS-DiT avoids multi-view video supervision by constructing a pseudo 4D Gaussian field directly from monocular videos using dense 3D point tracking. Its D3D-PT tracker is trained on 11,000 24-frame RGB-D sequences generated in Kubric at $r \in \mathbb{R}^{F \times H \times W \times 1}$3 resolution for 500,000 iterations on 8× NVIDIA A100 80GB, while GS-DiT itself is trained on 400K WebVid-10M clips for 100,000 steps at $r \in \mathbb{R}^{F \times H \times W \times 1}$4 (Bian et al., 5 Jan 2025). AutoWeather4D is notable for relying heavily on pretrained components and analytical rendering rather than on bespoke paired weather-conversion supervision: Pi3 for 4D reconstruction, DiffusionRenderer for inverse and forward rendering, OWL-ViT and SAM2 for semantics, Grounded-SAM for sky masking, WAN-FUN for refinement, and sparse LiDAR or a camera-height prior for depth calibration (Liu et al., 27 Mar 2026).

FrameDiffuser adopts a different deployment assumption: environment-specific specialization. It trains separate models for six Unreal Engine 5 environments—Electric Dreams, City Sample, Hillside Sample Project, Downtown West, City Park, and Derelict Corridor—using consecutive frame pairs with corresponding G-buffers (Beisswenger et al., 18 Dec 2025). The method explicitly prioritizes environment-specific consistency and inference speed over broad generalization.

These strategies indicate that G-buffer-guided video generation is not tied to a single supervision regime. Fully supervised synthetic rendering, pseudo-label bootstrapping, realistic artifact simulation, per-domain specialization, and analytical scene editing all coexist, with the chosen regime reflecting whether the task is relighting, refinement, weather conversion, object editing, or interactive rendering.

6. Applications, empirical behavior, and limitations

The application space is unusually broad because explicit scene buffers expose semantically meaningful controls. DiffusionRenderer demonstrates relighting, material editing, and realistic object insertion from a single video input (Liang et al., 30 Jan 2025). On forward rendering with ground-truth G-buffers and lighting, it achieves 26.0 / 0.780 / 0.201 PSNR/SSIM/LPIPS on SyntheticScenes, compared to 18.5 / 0.645 / 0.302 for RGB$r \in \mathbb{R}^{F \times H \times W \times 1}$5X and 20.7 / 0.630 / 0.300 for DiLightNet, and reaches 28.3 / 0.935 / 0.048 on SyntheticObjects (Liang et al., 30 Jan 2025). On relighting, it reports 27.50 / 0.918 / 0.067 on SyntheticObjects and 24.63 / 0.756 / 0.257 on SyntheticScenes (Liang et al., 30 Jan 2025).

GaussFusion targets reconstruction repair rather than relighting or editing. On DL3DV, its full model reaches PSNR 22.548, SSIM 0.832, LPIPS 0.278, and FID 3.933; on RE10K it reaches 28.652, 0.944, 0.180, and 6.672 (Zhu et al., 26 Mar 2026). Its modality ablation is especially informative for the broader topic: RGB-only conditioning gives 19.148 PSNR, 0.719 SSIM, 0.385 LPIPS, and 15.451 FID, while the full five-modality GP-Buffer gives 20.753, 0.773, 0.329, and 6.724, showing that every additional channel helps (Zhu et al., 26 Mar 2026).

AutoWeather4D shows how explicit intrinsic buffers can support physically controlled weather conversion. Evaluated on 120 Waymo Open Dataset scenes and 27,360 frames, it reports CLIP score 0.2586, Vehicle 3D detection IoU 0.915, Vehicle CLIP cosine similarity 0.871, and Human evaluation 0.826, with depth si-RMSE 0.247, Edge F1 0.129, FVD 886.8, and DOVER 0.370 (Liu et al., 27 Mar 2026). The paper emphasizes correct physical edits—removing inherited sunny shadows, producing localized headlight and streetlight effects, synthesizing snow accumulation and road ripples, and preserving dynamic vehicle structure—over pure perceptual scores (Liu et al., 27 Mar 2026).

FrameDiffuser addresses interactive neural forward rendering. Averaged across validation sets from its training distributions, it improves over X$r \in \mathbb{R}^{F \times H \times W \times 1}$6RGB from RGB$r \in \mathbb{R}^{F \times H \times W \times 1}$7X from 8.60 / 0.3566 / 0.5150 to 18.34 / 0.6377 / 0.2129 in PSNR/SSIM/LPIPS (Beisswenger et al., 18 Dec 2025). Its supplementary 24-frame comparison against DiffusionRenderer reports 20.96 / 0.6378 / 0.2030, compared with 13.03 / 0.4683 / 0.4583 (Beisswenger et al., 18 Dec 2025). The method can roll out over hundreds to thousands of frames, but it remains at approximately 1 frame per second on an RTX 4090 with 10 denoising steps and DPMSolver, so it is not yet real-time (Beisswenger et al., 18 Dec 2025).

Several limitations recur across the literature. DiffusionRenderer remains tied to the quality of predicted inverse-rendering buffers and estimated real-world lighting, can introduce slight variations in color or texture during editing, and is relatively expensive and offline rather than interactive (Liang et al., 30 Jan 2025). GaussFusion depends on having a usable initial 3DGS, degrades on rapid motion or severe motion blur because the VAE struggles to encode clean latents, and does not guarantee long-term coherence beyond its windowed strategy (Zhu et al., 26 Mar 2026). AutoWeather4D notes failure cases involving extreme long-tail dynamics such as vehicle splash, sensitivity to external feed-forward estimators, and the lack of an emissive channel, which causes self-illuminating objects like traffic lights to darken incorrectly when global illumination is reduced (Liu et al., 27 Mar 2026). FrameDiffuser sacrifices broad generalization through environment specialization, shows out-of-distribution style drift, inherits temporal flicker from the Stable Diffusion 1.5 VAE, and has no explicit long-horizon reset mechanism (Beisswenger et al., 18 Dec 2025).

Taken together, these systems indicate that G-buffer-guided video generation is best understood not as a single architecture, but as a family of structured-conditioning strategies for controllable video synthesis. The common thread is the use of rasterized or render-derived scene state to restrict the generative search space. The main axes of variation are the exact buffer representation, the conditioning pathway, whether generation is offline or causal, and whether the model learns a neural approximation to rendering, a refinement operator over imperfect renders, or a physically grounded editor whose diffusion stage is deliberately constrained.