DiT-Based Optical-Flow Discriminator

• The paper shows that the DiT-based optical-flow discriminator improves motion dynamics in few-step diffusion models using multi-scale transformer attention and adversarial loss.
• It fuses dense optical flow fields with per-pixel magnitude, enabling robust discrimination between genuine and synthesized motion for enhanced temporal coherence.
• Empirical results demonstrate higher motion scores and smoother dynamics in MoGAN, underscoring the importance of explicit motion supervision in video generation.

A DiT-based optical-flow discriminator is a motion-centric architecture incorporated in video diffusion post-training frameworks to distinguish genuine from synthesized video motion by analyzing dense optical flow fields, rather than relying on pixel or frame-level visual signals alone. In the MoGAN framework, this design is tightly coupled with adversarial learning and distribution-matching regularization to enable few-step diffusion models to generate video with substantially improved motion dynamics and coherence, without sacrificing spatial fidelity or prompt-conditioning.

1. Input Representation and Preprocessing

Given a video clip $x \in \mathbb{R}^{T \times C \times H \times W}$ (where $T$ is the frame count, $C$ the channel count, $H$ and $W$ the spatial dimensions), dense optical flows $o_{\text{raw}}$ are computed between successive frames using a frozen RAFT network $\mathcal{F}$, resulting in $$o_{\text{raw}} = \mathcal{F}(x) \in \mathbb{R}^{(T-1)\times2\times H\times W}$$. The last flow is duplicated to achieve temporal alignment $\hat{o} \in \mathbb{R}^{T\times2\times H\times W}$. The per-pixel magnitude $m(t,x,y) = \|\hat{o}(t,x,y)\|_2$ is computed and concatenated as a third channel, forming $$o = \text{concat}(\hat{o}, m) \in \mathbb{R}^{T\times 3 \times H \times W}$$. This design provides the discriminator with both directional and magnitude cues over time.

2. Discriminator Architecture: DiT Backbone and Multi-Scale Prediction

The discriminator’s backbone uses the first 16 transformer layers from the “Wan2.1-T2V-1.3B” Scalable Diffusion Transformer (DiT). Each layer implements standard multi-head self-attention across spatio-temporal patches, capturing motion at various temporal and spatial scales. To exploit hierarchical representations, lightweight “P-branches” are inserted at DiT layers $\{7,13,15\}$. Each P-branch receives its layer’s features, injects a learned “motion token,” and applies cross-attention between this token and the features. Outputs traverse a 2-layer MLP, and the outputs from all P-branches are concatenated and passed through a final MLP, yielding the scalar realness score $D_\phi(o) \in \mathbb{R}$.

For conditioning, the discriminator is prompt-aware; however, the prompt embedding is fixed to $c^* = \text{``a video with good motion''}$ and the diffusion timestep to $t^* = 0$, ensuring the discriminator focuses uniformly on motion quality irrespective of varied prompts or diffusion step inputs. Memory-efficient decoding for flow extraction is realized via truncated BPTT, unrolling only 12 of 21 latent chunks and detaching the recurrent state after each window.

3. Input/Output Formats and Computational Workflow

Input Shape	Data Type	Output
$T\times 3\times H\times W$	Optical flow + per-pixel magnitude	Scalar $D_\phi(o)$

The discriminator receives a tensor representing $T$ frames with 3 channels (2D flow + magnitude) and outputs a single scalar discriminating real versus generated motion. This abstraction enforces the notion that motion statistics, not framewise content, are the critical signals for adversarial supervision.

4. Loss Functions and Regularization

Adversarial training deploys a logistic GAN loss augmented with R1 (real-flow) and R2 (synthetic-flow) regularizers:

• Discriminator loss:

$$\mathcal{L}_{\text{GAN}}^\phi = \mathbb{E}_{t,c}[\,g(-D_\phi(o^{\text{real}})) + g(D_\phi(o^{\text{gen}}))\,], \ \text{with} \ g(x) = \log(1 + e^{x})$$

• R1/R2 regularizers:

$$\mathcal{L}_{R1}^\phi = \|D_\phi(o^{\text{real}}) - D_\phi(o^{\text{real}} + \epsilon)\|_2^2 \ \mathcal{L}_{R2}^\phi = \|D_\phi(o^{\text{gen}}) - D_\phi(o^{\text{gen}} + \epsilon)\|_2^2$$

• Generator (motion-GAN) adversarial loss:

$$\mathcal{L}_{\text{GAN}}^\theta = \mathbb{E}_{t,c}[\,g(-D_\phi(o^{\text{gen}}))\,]$$

• Combined MoGAN objective:

$$\mathcal{L}_{\text{MoGAN}} = \lambda_1 \mathcal{L}_{\text{GAN}}^\theta + \lambda_2 \mathcal{L}_{\text{GAN}}^\phi + \lambda_{R1}\mathcal{L}_{R1}^\phi + \lambda_{R2}\mathcal{L}_{R2}^\phi$$

Joint optimization with a distribution-matching (DMD) regularizer preserves spatial fidelity and text/condition alignment:

• Generator KL objective:

$$\mathcal{L}_{\text{DMD}} = \mathbb{E}_{t}[\,D_{\mathrm{KL}}(p_t^{\text{real}} \,\|\, p_t^{\text{gen}})\,]$$

• The velocity-parameterization implements this as a flow-matching gradient,
• A critic-side “fake score” regression loss $\mathcal{L}_{\text{fake}}^\phi$ further anchors the generator to teacher dynamics.

This loss structure is calibrated to prevent adversarial-induced mode collapse or image quality drift observed when using GAN-only supervision.

5. Integration into Few-Step Diffusion Model Post-Training

MoGAN applies post-training to a distilled 3-step Wan2.1-T2V-1.3B video diffusion generator. The workflow is:

1. Warm-up the generator under $\mathcal{L}_{\text{DMD}}$ to ensure reliable optical flows.
2. For each training iteration:
   • Update generator parameters $\theta$ by $\nabla_\theta \mathcal{L}_{\text{DMD}}$
   • Update critic fake-score $\phi$ via $\mathcal{L}_{\text{fake}}^\phi$ (4$\times$ per generator step)
   • Decode video clips at selected steps, compute $o^{\text{greg}}, o^{\text{real}}$
   • Update discriminator $\phi$ via GAN and R1/R2 regularizers
   • Update generator $\theta$ via $\nabla_\theta \mathcal{L}_{\text{GAN}}^\theta$

Optimization uses AdamW with a learning rate of $1 \times 10^{-5}$, loss weights $\lambda_1 = \lambda_2 = 0.5$ (GAN), $\lambda_{R1} = \lambda_{R2} = 0.3$, batch sizes 64 (discriminator) and 16 (generator), and R1/R2 noise $\sigma=0.01$.

6. Empirical Performance and Ablation Outcomes

On VBench, MoGAN outperforms both its 50-step teacher and 3-step DMD-only distilled model in motion metrics:

Model	Smoothness (%)	Dynamics	Motion Score
Wan2.1 (50-step)	98.0	0.83	0.905
DMD-only (3-step)	98.8	0.73	0.859
MoGAN (3-step)	98.6	0.96	0.973

On VideoJAM-Bench, MoGAN increases motion score and dynamics by +7.4% over the teacher, with equivalent aesthetics and image quality. Human preference studies (148 videos) report MoGAN preferred for motion quality (52% vs. 38% for teacher, 56% vs. 29% for DMD).

Key ablations:

• Removing DMD regularization ($\mathcal{L}_{\text{DMD}}$) leads to mode collapse (dynamics $0.35$, motion score $0.674$).
• Dropping R1/R2 regularizers results in decreased smoothness (95.1%), indicating unstable adversarial training.
• Adversarial learning on pixel space (no flow) raises dynamics modestly ($0.85$) but does not match gains from flow-based discrimination.

This suggests dense flow-based adversarial objectives directly incentivize multi-frame coherence not captured in per-frame or pixel-difference spaces.

7. Significance and Context in Video Generation Models

The DiT-based optical-flow discriminator in MoGAN demonstrates the utility of flow-centric, transformer-powered discriminators for adjudicating video motion realism in generative models. By learning multi-scale, prompt-agnostic representations of motion, this framework addresses key weaknesses in diffusion models—namely, frame-level sharpness without plausible dynamics—while empirically preserving or improving visual fidelity and efficiency. A plausible implication is that explicit motion supervision is pivotal for scalable, fast, and robust video generation—the technique bridges the gap between adversarial sharpness enhancements and the need for reliable, temporally consistent motion in conditioned video synthesis models (Xue et al., 26 Nov 2025).