FVGen: Fast Video Diffusion Distillation

• FVGen is a framework for accelerating diffusion-based novel-view synthesis by distilling a high-quality multi-step teacher model into a four-step student model.
• It employs a two-stage distillation process with GAN-based initialization followed by softened reverse KL divergence minimization to achieve over 90% speedup.
• Empirical evaluations on datasets like MipNeRF360 and Tanks-and-Temples demonstrate comparable visual quality while significantly reducing inference time.

Searching arXiv for FVGen and related novel-view synthesis/video diffusion papers. Searching arXiv for "FVGen Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation". FVGen, short for “Fast Video Generation,” is a framework for accelerating diffusion-based novel-view synthesis from sparse input images by distilling a high-quality multi-step Video Diffusion Model (VDM) into a few-step student model. It is introduced in “FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation” (Teng et al., 8 Aug 2025). The method targets a central bottleneck in sparse-view 3D reconstruction pipelines: although latent VDMs can generate dense intermediate observations that improve spatial coverage, their standard sampling procedures are slow. FVGen addresses this by enabling fast novel-view synthesis in as few as four denoising steps, using a distillation procedure that combines GAN-based student initialization with softened reverse KL-divergence minimization, and reports more than 90% sampling-time reduction while maintaining similar, and in some cases better, visual quality on standard metrics (Teng et al., 8 Aug 2025).

1. Problem setting and objectives

FVGen is formulated for novel-view synthesis from sparse inputs, a setting described as severely ill-posed when only fewer than two or three images of a scene are available (Teng et al., 8 Aug 2025). In this regime, recent methods use latent VDMs conditioned on sparse 3D cues such as point clouds or sparse NeRF renders to hallucinate dense multi-view video sequences with spatial and temporal consistency. The limitation identified by FVGen is not the viability of VDM-based synthesis itself, but the latency of standard diffusion sampling, which can require 100–1000+ denoising steps and thereby impede downstream tasks such as dynamic 3D reconstruction and real-time rendering (Teng et al., 8 Aug 2025).

The framework’s stated objective is to distill the conditional distribution $p_{\rm teacher}(\mathbf{x}\!\mid\!c)$ of a multi-step teacher into a few-step student distribution $p_{\rm student}(\mathbf{x}\!\mid\!c)$, so that the student can generate $\mathbf{x}\sim p_{\rm student}$ in only four denoising iterations while matching the teacher’s sample quality on FID, PSNR, SSIM, and LPIPS (Teng et al., 8 Aug 2025). In the paper’s framing, this is not merely an acceleration heuristic, but a distribution-matching problem over conditional video synthesis.

A notable practical emphasis is downstream reconstruction under sparse input views greater than two, where pre-trained VDMs must often be run multiple times to achieve better spatial coverage (Teng et al., 8 Aug 2025). This positions FVGen within a broader class of methods that treat generative multi-view synthesis as a precursor to 3D reconstruction rather than as an end in itself.

2. Teacher–student formulation

The teacher in FVGen is the latent video diffusion model behind ViewCrafter, with video clips represented in latent space after VAE encoding (Teng et al., 8 Aug 2025). For a video clip $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$, the latent representation is $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$. The teacher’s forward noising process is

$$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$

and a UNet-based network $\boldsymbol{\epsilon}_\phi(\mathbf{z}_t,t,c)$ is trained by the MSE objective

$$\mathcal{L}_{\rm diff}^{\rm teach} = \mathbb{E}_{t,\mathbf{z}_0,\epsilon}\bigl\|\epsilon - \epsilon_\phi(\mathbf{z}_t,t,c)\bigr\|^2.$$

At inference, the teacher performs $T$ reverse-diffusion denoising steps conditioned on sparse inputs $c=(I^{\rm obs},\pi^{\rm obs},\pi^{\rm tgt})$ to sample novel-view sequences (Teng et al., 8 Aug 2025).

The student model $p_{\rm student}(\mathbf{x}\!\mid\!c)$0 shares the same UNet backbone and conditioning mechanism as the teacher, but denoises in only $p_{\rm student}(\mathbf{x}\!\mid\!c)$1 steps, with $p_{\rm student}(\mathbf{x}\!\mid\!c)$2 in practice (Teng et al., 8 Aug 2025). Its forward noising and reverse sampling preserve the same functional form, but use a reduced schedule $p_{\rm student}(\mathbf{x}\!\mid\!c)$3. The distillation target is expressed as

$p_{\rm student}(\mathbf{x}\!\mid\!c)$4

where $p_{\rm student}(\mathbf{x}\!\mid\!c)$5 is the teacher’s mean prediction and $p_{\rm student}(\mathbf{x}\!\mid\!c)$6 is the VAE decoder (Teng et al., 8 Aug 2025).

This teacher–student construction is important because FVGen does not alter the conditioning interface or the basic latent-video architecture. Instead, it compresses the denoising trajectory. A plausible implication is that FVGen is designed to preserve compatibility with an existing VDM pipeline while reducing inference depth.

3. Adversarial video diffusion distillation

FVGen’s core methodological contribution is a two-stage distillation procedure: GAN-based student initialization followed by softened reverse-KL-based distribution matching (Teng et al., 8 Aug 2025). The paper explicitly motivates this design by noting that direct application of Distribution Matching Distillation when compressing from $p_{\rm student}(\mathbf{x}\!\mid\!c)$7 to $p_{\rm student}(\mathbf{x}\!\mid\!c)$8 steps can collapse.

In the initialization stage, the student is treated as a generator $p_{\rm student}(\mathbf{x}\!\mid\!c)$9 in an adversarial game against a discriminator $\mathbf{x}\sim p_{\rm student}$0 built from the teacher’s UNet features. For random latents $\mathbf{x}\sim p_{\rm student}$1 and time step $\mathbf{x}\sim p_{\rm student}$2, the student produces a fake noisy video $\mathbf{x}\sim p_{\rm student}$3. The discriminator taps mid-layer features of the frozen teacher UNet and optimizes

$\mathbf{x}\sim p_{\rm student}$4

$\mathbf{x}\sim p_{\rm student}$5

According to the paper, this yields a stronger initialization than regression on ODE-solved latents (Teng et al., 8 Aug 2025).

The second stage replaces plain reverse KL with a softened variant. Standard DMD minimizes

$\mathbf{x}\sim p_{\rm student}$6

with gradient proportional to the difference of score functions,

$\mathbf{x}\sim p_{\rm student}$7

FVGen instead uses

$\mathbf{x}\sim p_{\rm student}$8

which the paper describes as preserving mode-seeking behavior while penalizing neglect of minor modes (Teng et al., 8 Aug 2025). Writing the density ratio as $\mathbf{x}\sim p_{\rm student}$9, the objective becomes

$$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$0

with gradient

$$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$1

FVGen reuses the discriminator from the GAN stage to approximate the density ratio by

$$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$2

During this phase the student also updates its diffusion-mean predictor through

$$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$3

which keeps $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$4 synchronized with $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$5 (Teng et al., 8 Aug 2025). The paper reports that the combined schedule, with relative weight $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$6 for $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$7, yields stable convergence and avoids mode collapse (Teng et al., 8 Aug 2025).

4. Training procedure and inference profile

FVGen’s training algorithm is explicitly two-phase. First, $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$8 and $$\mathbf{x}_0\in\mathbb{R}^{F\times C\times H\times W}$$9 are initialized from the teacher $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$0. Second, a GAN stage of approximately 4K iterations samples real video $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$1, noises it to $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$2, generates $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$3, and updates $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$4 and $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$5. Third, a DMD stage of approximately 5K iterations computes the soften-RKL gradient together with the diffusion MSE loss to update $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$6 and $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$7 (Teng et al., 8 Aug 2025).

At inference, the student performs only $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$8 denoising steps at times $$\mathbf{z}_0=\mathcal{E}(\mathbf{x}_0)\in\mathbb{R}^{F\times C\times h\times w}$$9, starting from $$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$0 (Teng et al., 8 Aug 2025). The supplementary details specify uniformly spaced student timesteps in $$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$1, input resolution $$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$2, training on 8$$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$3NVIDIA H100 GPUs with batch size 4, AdamW at learning rate $$q(\mathbf{z}_t\mid \mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\sqrt{\alpha_t}\,\mathbf{z}_{t-1},\,(1-\alpha_t)\,\mathbf{I}\bigr),\quad t=1,\dots,T,$$4, and a total training time of 1 day (Teng et al., 8 Aug 2025).

The paper reports two distinct runtime comparisons. In the algorithmic description, inference produces a full novel-view video in approximately 1 s, compared with approximately 13 s for the 100-step teacher (Teng et al., 8 Aug 2025). In the 56-view synthesis evaluation, wall-clock time is reported as 5.1 s versus 66.3 s on MipNeRF360 and 5.0 s versus 65.9 s on Tanks-and-Temples (Teng et al., 8 Aug 2025). This suggests that the shorter timing refers to a particular generation unit, while the longer timing reflects an end-to-end evaluation configuration; the paper does not further reconcile the two measurements in the provided summary.

A common misconception would be to interpret FVGen as a one-stage adversarial replacement for diffusion. The training design indicates otherwise: the GAN is used for initialization, while the student remains a diffusion model with a reduced denoising schedule and a retained diffusion loss (Teng et al., 8 Aug 2025).

5. Empirical evaluation

FVGen is evaluated on MipNeRF360 and Tanks-and-Temples using PSNR, SSIM, LPIPS, FID, and wall-clock time (Teng et al., 8 Aug 2025). The principal comparison is a 4-step FVGen student against ViewCrafter with 100+ steps for 56-view synthesis.

Dataset	FVGen (4-step)	ViewCrafter (100+ steps)
MipNeRF360 PSNR	16.28	16.35
MipNeRF360 SSIM	0.352	0.346
MipNeRF360 LPIPS	0.429	0.433
MipNeRF360 FID	17.44	16.28
MipNeRF360 Time	5.1 s	66.3 s
Tanks-and-Temples PSNR	18.72	18.69
Tanks-and-Temples SSIM	0.411	0.402
Tanks-and-Temples LPIPS	0.210	0.208
Tanks-and-Temples FID	23.64	23.94
Tanks-and-Temples Time	5.0 s	65.9 s

On MipNeRF360, FVGen records PSNR 16.28 versus 16.35, SSIM 0.352 versus 0.346, LPIPS 0.429 versus 0.433, FID 17.44 versus 16.28, and Time 5.1 s versus 66.3 s, corresponding to approximately 92% speedup (Teng et al., 8 Aug 2025). On Tanks-and-Temples, it reports PSNR 18.72 versus 18.69, SSIM 0.411 versus 0.402, LPIPS 0.210 versus 0.208, FID 23.64 versus 23.94, and Time 5.0 s versus 65.9 s (Teng et al., 8 Aug 2025). The overall interpretation given by the paper is that the method generates the same number of novel views with similar, or sometimes better, visual quality while substantially reducing sampling time.

The comparison against distillation baselines is also central. Relative to ViewCrafter+DMD2 and ViewCrafter+CausVid, FVGen is reported to achieve superior PSNR, SSIM, LPIPS, and FID by 0.5–1.5 dB-equivalent, with stable training and no mode collapse (Teng et al., 8 Aug 2025). The ablation studies further state that GAN initialization is essential, that without it the student fails to match teacher modes, that soften-RKL is more stable than plain reverse-KL, and that the combined GAN + DMD + Soften-RKL configuration yields the best metrics (Teng et al., 8 Aug 2025).

These results define FVGen’s empirical contribution narrowly but clearly: it is a distillation scheme for preserving fidelity under aggressive step compression. It is not presented as improving the teacher’s underlying reconstruction prior, only as making that prior operationally faster.

6. Scope, limitations, and nomenclature

FVGen currently relies on a pre-trained teacher and a moderate dataset of multi-view videos, DL3DV-10K (Teng et al., 8 Aug 2025). The paper states that performance may degrade if input views are extremely sparse, specifically below 2, or if scene types diverge strongly from the training distribution (Teng et al., 8 Aug 2025). It also notes a quality–latency trade-off: the four-step schedule exchanges a small FID regression of approximately 1 point for a large speedup, and adaptive schedules such as 6–8 steps are proposed as a future direction (Teng et al., 8 Aug 2025).

Other future directions identified in the paper include integrating explicit 3D geometry during distillation, distilling into non-diffusive architectures such as Gaussiansplatting, and extending the framework to free-view video editing tasks (Teng et al., 8 Aug 2025). These directions indicate that FVGen is presently a distillation framework around a latent VDM rather than a general-purpose acceleration principle for all view-synthesis models.

The term “FVGen” also has a potential naming ambiguity outside this paper. In “Neural Functionally Generated Portfolios” (Monoyios et al., 24 Jun 2025), “FVGen” appears only as a query label suggesting a possible extension toward a fully functional generator over paths or features beyond market weights, rather than as the name of the portfolio framework itself. In that context, the paper states that “The FVGen query suggests a desire for a fully functional (infinite-dimensional) generator over paths or features beyond market weights” (Monoyios et al., 24 Jun 2025). This is unrelated to the novel-view-synthesis framework introduced in (Teng et al., 8 Aug 2025). For the term as an established research system name, the relevant usage is the video-diffusion distillation method for fast novel-view synthesis.

Within the literature summarized here, FVGen is therefore best understood as a specific 2025 framework for few-step latent video diffusion in sparse-view novel-view synthesis, characterized by a teacher–student architecture, GAN-based initialization, soften reverse-KL distribution matching, four-step inference, and more than 90% sampling-time reduction with comparable perceptual quality (Teng et al., 8 Aug 2025).