Align4Gen: Aligned Video Diffusion Framework

• Align4Gen is a training framework for video diffusion models that fuses multi-encoder features to enhance discriminability and temporal consistency.
• It aligns intermediate V‑DiT tokens with pre-trained vision encoder outputs via a cosine-based loss, improving metrics such as FVD, FID, and IS.
• The framework is objective-agnostic, integrates multi-feature fusion, and accelerates convergence while maintaining high visual fidelity across datasets.

Searching arXiv for “Align4Gen” and closely related alignment-based generative modeling papers to ground the article. {"query":"Align4Gen arXiv alignment generative video diffusion", "max_results": 10} Align4Gen is a training-time framework for video diffusion transformers that aligns intermediate latent tokens of a V‑DiT with feature representations from multiple pre-trained vision encoders, fuses complementary encoder features such as DINOv2 and SAM2.1 Hiera, and adds a feature-alignment loss on top of the usual diffusion or flow-matching loss. Its stated objective is to improve the feature representation quality of video generators by importing the discriminability and temporal consistency of self-supervised vision encoders into the internal feature space of a video diffusion model, thereby improving video quality metrics such as FVD, FID, IS, and CLIP while accelerating convergence (Lee et al., 11 Sep 2025).

1. Definition and conceptual scope

Align4Gen is defined as a framework for training video diffusion models, especially Video Diffusion Transformers, through multi-feature fusion and alignment from self-supervised vision encoders. The central claim is that feature representation quality in video generators has been underexploited relative to architectural changes such as diffusion transformers and training-objective changes such as flow matching, even though diffusion-model features remain inferior to strong self-supervised encoders such as DINOv2 or MAE in discriminability and temporal stability (Lee et al., 11 Sep 2025).

The method is explicitly training-only. During training, Align4Gen extracts per-frame patch features from multiple pre-trained image encoders, normalizes and concatenates them, maps intermediate V‑DiT tokens through a small MLP into the encoder feature space, and minimizes a cosine-based projection loss. At inference, the video generator is used as usual; the auxiliary MLPs and encoders are not needed (Lee et al., 11 Sep 2025).

A key conceptual point is that Align4Gen is objective-agnostic with respect to the underlying generative loss. It is implemented on top of both the standard diffusion $\epsilon$-prediction loss and rectified flow or flow matching loss, with ablations reporting gains under both settings (Lee et al., 11 Sep 2025). This positions the method as a supervisory layer over existing V‑DiT training recipes rather than as a replacement for the base generative objective.

2. Analytical basis: IICR and encoder selection

A defining contribution of Align4Gen is its systematic analysis of which pre-trained encoders are suitable for feature alignment in video diffusion training. The paper evaluates DINOv2, SAM 2.1 Hiera image encoder, VideoMAE, DUSt3R, and additional encoders such as Stable Diffusion 3 transformer features and the vision encoder of InternVL, then introduces the Intra–Inter Consistency Ratio, or IICR, to assess both discriminability and temporal consistency (Lee et al., 11 Sep 2025).

For each video, patch features $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$ are extracted across frames, K-means is run over all patch features from all frames, and cluster centroids $\mathbf{u}_i$ and within-cluster standard deviations $\sigma_i$ are computed. The inter-cluster distance is defined as

$$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$

the intra-cluster distance as

$D_{\text{intra}} = \max_i \sigma_i,$

and the ratio as

$$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$

High IICR indicates strong semantic separation and stable representations over time (Lee et al., 11 Sep 2025).

The reported empirical findings are specific. DINOv2 and SAM2.1 Hiera show consistently high IICR over varying $K$, and PCA visualizations show stable colors or embeddings for the same foreground and background regions across frames. By contrast, VideoMAE and DUSt3R show IICR degradation as $K$ increases, and PCA indicates unstable patch embeddings across frames. The interpretation given in the source is that temporal compression and aggregation in VideoMAE hurt frame-wise token stability, while DUSt3R is geometry-focused and not optimized for temporal feature consistency in 2D appearance (Lee et al., 11 Sep 2025).

This analysis underwrites the method’s encoder choice. Align4Gen selects DINOv2 and SAM2.1 Hiera as the primary alignment targets because they provide strong IICR, complementary frequency characteristics, and efficient computation compared with heavier encoders such as SD3 or InternVL (Lee et al., 11 Sep 2025).

3. Multi-feature fusion and alignment mechanism

The alignment mechanism combines frequency analysis, feature fusion, and patch-wise projection loss. For a feature map $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$, the paper computes a log-magnitude Fourier spectrum

$\mathbf{F} \in \mathbb{R}^{h \times w \times c}$0

defines the DC component as $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$1, and defines the high-frequency region

$\mathbf{F} \in \mathbb{R}^{h \times w \times c}$2

The frequency-difference metric is

$\mathbf{F} \in \mathbb{R}^{h \times w \times c}$3

The reported result is that $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$4 differs by $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$5 in log scale between DINOv2 and SAM2.1 Hiera, approximately a $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$6 ratio in magnitude, with DINOv2 emphasizing low-frequency semantics and SAM2.1 Hiera emphasizing high-frequency details (Lee et al., 11 Sep 2025).

Feature fusion is implemented by L2-normalizing each encoder’s patch features along the channel dimension and concatenating them:

$\mathbf{F} \in \mathbb{R}^{h \times w \times c}$7

The alignment target is therefore a multi-encoder, multi-frequency feature for each patch and frame (Lee et al., 11 Sep 2025).

Intermediate V‑DiT tokens $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$8 are then mapped with a shared MLP,

$\mathbf{F} \in \mathbb{R}^{h \times w \times c}$9

and aligned to the fused target with cosine distance:

$\mathbf{u}_i$0

averaged over matched tokens and samples (Lee et al., 11 Sep 2025). The paper also tests a two-MLP formulation, one per encoder, but reports that concatenation plus a single MLP performs better, which it attributes to the unified target space encouraging the generator tokens to capture both semantic and detail information simultaneously (Lee et al., 11 Sep 2025).

Temporal structure is handled implicitly rather than through an explicit temporal penalty. Encoder features are computed frame-wise, but high-IICR image encoders yield temporally stable targets for the same object across frames, so aligning V‑DiT tokens at all frames to those targets is intended to encourage temporal consistency in the generator’s internal representations (Lee et al., 11 Sep 2025).

4. Integration with Video Diffusion Transformers

Align4Gen is built on a V‑DiT architecture similar to Latte. A 2D VAE, specifically the Stable Diffusion v1.4 VAE, encodes each frame independently into latents; latent frames are patch-embedded with patch size $\mathbf{u}_i$1; and the transformer contains spatial and temporal sub-blocks, with spatial self-attention across spatial tokens within each frame and temporal self-attention across time for each patch location (Lee et al., 11 Sep 2025).

Alignment is applied to intermediate tokens from a selected block. For V‑DiT‑XL, the reported experiments indicate that mid-depth, such as depth $\mathbf{u}_i$2 for an $\mathbf{u}_i$3 model, works slightly better, and features are taken after the spatial transformer sub-block. The paper reports that spatial alignment is slightly better than temporal alignment, and interprets mid-level spatial features as sufficiently semantic without being overly entangled by temporal mixing (Lee et al., 11 Sep 2025).

The total training loss is

$\mathbf{u}_i$4

where $\mathbf{u}_i$5 balances the alignment term (Lee et al., 11 Sep 2025). For diffusion training, the base loss is

$\mathbf{u}_i$6

For rectified flow or flow matching, the paper uses

$\mathbf{u}_i$7

The method is therefore presented as compatible with both major training paradigms used in V‑DiT systems (Lee et al., 11 Sep 2025).

The main experimental configurations are concrete: UCF‑101 for class-conditional generation, SkyTimelapse and FaceForensics for unconditional generation, all resized to $\mathbf{u}_i$8 with $\mathbf{u}_i$9 frames; V‑DiT‑L has $\sigma_i$0 blocks and about $\sigma_i$1M parameters; V‑DiT‑XL has $\sigma_i$2 blocks and about $\sigma_i$3M parameters; DDIM with $\sigma_i$4 steps is used for diffusion-based models and Euler with $\sigma_i$5 steps for flow-based models (Lee et al., 11 Sep 2025).

5. Empirical performance and ablations

The empirical evaluation uses content-debiased FVD based on VideoMAE features as the main metric, alongside frame-level FID, Inception Score on UCF‑101, and frame-wise CLIP similarity:

$\sigma_i$6

The FVD definition reported is

$\sigma_i$7

and the UCF‑101 Inception Score is

$\sigma_i$8

These definitions are given directly in the source description (Lee et al., 11 Sep 2025).

On UCF‑101 with V‑DiT‑L and diffusion loss, the paper reports the following results.

Setting	FVD	FID
Baseline at 400K	262.77	43.87
Fusion at 400K	206.73	38.89
Baseline at 1M	221.63	40.70
Fusion at 1M	187.46	37.31

The same table also reports IS and CLIP. At $\sigma_i$9K steps, baseline V‑DiT has IS $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$0 and CLIP $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$1, while Fusion has IS $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$2 and CLIP $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$3. At $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$4M steps, baseline has IS $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$5 and CLIP $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$6, while Fusion has IS $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$7 and CLIP $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$8 (Lee et al., 11 Sep 2025). The paper explicitly notes that at $$D_{\text{inter}} = \min_{i \neq j} \|\mathbf{u}_i - \mathbf{u}_j\|,$$9K, Align4Gen-Fusion already beats the baseline at $D_{\text{intra}} = \max_i \sigma_i,$0M in both FVD and FID, which it interprets as at least $D_{\text{intra}} = \max_i \sigma_i,$1 faster convergence in terms of sample quality (Lee et al., 11 Sep 2025).

On unconditional generation, gains depend on dataset structure. For SkyTimelapse, improvements are modest: at $D_{\text{intra}} = \max_i \sigma_i,$2K iterations, baseline yields FVD $D_{\text{intra}} = \max_i \sigma_i,$3, FID $D_{\text{intra}} = \max_i \sigma_i,$4, and CLIP $D_{\text{intra}} = \max_i \sigma_i,$5, while Fusion yields FVD $D_{\text{intra}} = \max_i \sigma_i,$6, FID $D_{\text{intra}} = \max_i \sigma_i,$7, and CLIP $D_{\text{intra}} = \max_i \sigma_i,$8; at $D_{\text{intra}} = \max_i \sigma_i,$9K, baseline gives FVD $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$0, FID $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$1, and CLIP $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$2, while Fusion gives FVD $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$3, FID $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$4, and CLIP $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$5 (Lee et al., 11 Sep 2025). On FaceForensics, gains are larger: at $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$6K, baseline has FVD $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$7 and FID $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$8, while Fusion reaches FVD $$\text{IICR} = \frac{D_{\text{inter}}}{D_{\text{intra}}}.$$9 and FID $K$0; at $K$1K, baseline has FVD $K$2 and FID $K$3, while Fusion has FVD $K$4 and FID $K$5 (Lee et al., 11 Sep 2025).

Ablation studies reinforce the encoder-selection argument. On UCF‑101 with V‑DiT‑XL, $K$6K steps, rectified flow objective, and depth $K$7, the paper reports FVD $K$8 for DINOv2, $K$9 for SAM2, $K$0 for VideoMAE, $K$1 for DUSt3R, and $K$2 for SD3 features (Lee et al., 11 Sep 2025). This ordering is said to match IICR scores. Fusion with a single MLP performs better than two separate MLPs, with FVD $K$3 versus $K$4 (Lee et al., 11 Sep 2025). Alignment after the spatial block at depth $K$5 outperforms both depth $K$6 spatial alignment and depth $K$7 temporal alignment, with FVD $K$8, $K$9, and $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$0, respectively, against a no-alignment baseline of $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$1 (Lee et al., 11 Sep 2025). Under both diffusion and flow objectives, Fusion is best: for diffusion, baseline is $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$2 and Fusion $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$3; for flow, baseline is $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$4 and Fusion $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$5 (Lee et al., 11 Sep 2025).

The supplement comparison using standard I3D-based FVD reports Latte at $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$6, V‑DiT plus Fusion at $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$7, PVDM at $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$8, and Matten at $$\mathbf{F} = \mathcal{E}(x_0) \in \mathbb{R}^{h \times w \times c}$$9 (Lee et al., 11 Sep 2025). Qualitative descriptions in the source emphasize sharper human silhouettes, more consistent limb positions and shapes, less flicker, smoother space–time slices on UCF‑101, and clearer facial structure and expressions on FaceForensics (Lee et al., 11 Sep 2025).

6. Interpretation, limitations, and related usage of the name

The immediate interpretation of the reported results is that Align4Gen improves V‑DiT training when the supervisory encoders are both discriminative and temporally stable at the patch level. The paper’s own explanation is that DINOv2 contributes low-frequency semantics such as object identity and category, while SAM2.1 Hiera contributes high-frequency details such as edges, texture, and segmentation cues, and that aligning V‑DiT tokens to both improves the internal representations used by the generator (Lee et al., 11 Sep 2025).

The method has explicit limitations. The gains are dataset-dependent: on low-discriminability datasets such as SkyTimelapse, improvements are small. The source also reports a limitation in text-to-video fine-tuning: using PixArt-α image weights for spatial blocks and aligning to image encoders during text-to-video fine-tuning leads to early projection misalignment and eventual degradation, with the stated interpretation that the semantic prior is already strong and alignment disturbs it (Lee et al., 11 Sep 2025). Computational overhead is another constraint: heavy encoders such as InternVL may have reasonable IICR and competitive FVD up to $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$00K iterations, but are approximately $\mathbf{F} \in \mathbb{R}^{h \times w \times c}$01 slower than DINOv2 and therefore not practically attractive in the reported experiments (Lee et al., 11 Sep 2025).

Within related work, Align4Gen is described as related to REPA, which aligns image DiT features to DINOv2 or MAE, but extends the concept to video diffusion transformers while introducing IICR and multi-feature fusion. It is also contrasted with task-specific auxiliary-loss approaches such as Track4Gen, VideoJAM, GenRec, and JOG3R, and with CLIP-guidance or perceptual-loss methods that supervise output images rather than intermediate patch-space features (Lee et al., 11 Sep 2025). This suggests a narrower and more architectural interpretation of “alignment” than methods based on output-level semantic guidance.

A separate but related usage of the name appears in the X‑to‑4D literature. “Alignment Is All You Need For X‑to‑4D Generation” interprets “Align4Gen” as an alignment-based generative modeling philosophy in which multimodal priors are composed and reconciled through distances and optimization rather than learned end-to-end. In that setting, Align4D decomposes X‑to‑4D generation into geometry generation, motion generation, Object Distance Alignment, Motion–Geometry Joint Alignment, and Asynchronous Optimization (Miao et al., 2 Jul 2026). This suggests that “Align4Gen” can also function as a broader label for alignment-first generative system design, although the explicit framework named Align4Gen in the supplied literature is the video-diffusion training method of multi-feature fusion and encoder alignment (Lee et al., 11 Sep 2025).