Gen4U: Unifying Video Generation and Understanding via Diffusion

Abstract: Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.

• The paper introduces Gen4U, a unified video model that uses diffusion transformers to combine generation and understanding tasks via probing intermediate activations.
• By analyzing intermediate feature activations across noise levels, the study reveals a dual-axis progression that balances high-level semantic cues and low-level geometric details.
• Empirical evaluations on tasks like video classification, depth, and camera pose estimation demonstrate competitive performance with efficient, single-pass feature extraction.

Gen4U: Unifying Video Generation and Understanding via Diffusion

Motivation and Context

The paper presents Gen4U, a framework that unifies video generation and video understanding by leveraging large-scale diffusion models. Contemporary video representation paradigms, including contrastive methods and pixel-level reconstruction approaches, offer either high-level semantic or low-level geometric understanding, rarely both. Contrastive learning and VLMs operate predominantly in language space, often missing precise spatiotemporal cues, while masked autoencoding preserves geometry but fails to generalize to broad semantic tasks. Gen4U addresses this dichotomy, proposing a single task-agnostic video foundation model capable of both generation and understanding by systematically probing intermediate activations of diffusion transformers.

Analysis of Diffusion Representations

The investigation centers on the latent manifold of video diffusion models, notably Veo3 and Wan 2.2. Through the lens of recent mutual-kNN alignment metrics, the paper maps the evolution of representations across network depth and noise levels.

Multiple zero-shot probes are employed:

• PCA visualizations elucidate feature complexity, revealing that high noise levels encode low-frequency global semantics, while lower noise levels yield spatially distributed, high-frequency geometric details.

Figure 1: PCA visualization demonstrates how activations from Veo3 organize spatial tokens; moderate noise levels maximize semantic alignment.

• Mutual k-NN alignment quantifies semantic congruence between diffusion embeddings and reference encoders, showing that Veo3 achieves strong alignment not only with text captions but also with discriminative visual encoders such as DINOv2 and VideoMAEv2.

Figure 2: Veo3 and Wan 2.2's intermediate features align strongly with language embeddings and outperform prior diffusion models on zero-shot video-text alignment.

Figure 3: Diffusion activations align with strong image/video models and underlying scene semantics, suggesting latent space is broadly applicable.

These analyses reveal a structured, dual-axis development: intermediate activations become progressively more semantically rich as depth and moderate noise increase, with a consistent semantic bottleneck emerging at $\approx$ 60% noise.

Efficient Feature Extraction and Probing

Gen4U identifies optimal extraction points via both linear and attention-based probes. Linear probes extract global semantics at moderate noise and depth, while attention mechanisms are necessary to decode spatially scattered fine details at lower noise levels and deeper blocks.

Figure 4: Linear and attention classification probes peak at intermediate depths/noise, illustrating optimal points for extracting semantic-rich representations.

Empirical sweeps confirm that the sweet spot for both semantic and geometric tasks converges around depth 75-80% and noise 30-60%. This enables a single forward pass through the diffusion backbone, yielding computational efficiency comparable to discriminative encoders.

Downstream Task Evaluation

Semantic Understanding

Gen4U, employing frozen Veo3 representations paired with lightweight attention decoders, achieves strong results on a range of tasks:

• Video Classification: On the SSv2 dataset, Gen4U attains 71.3% top-1 accuracy with basic protocols, rising to 72.6% with in-feature data augmentation, approaching the best latent prediction baselines.
• Captioning: Attaching a Gemma-2-2B LLM as decoder, Gen4U's diffusion features generate reasonable captions for image and video datasets (COCO, SSv2, Vatex), although performance lags discriminative approaches like SigLIP (e.g., on COCO, Gen4U achieves 54.9 CIDEr vs. 118.5 for SigLIP), yet excels in action-centric video datasets.

Geometry Understanding

• Depth Estimation: On ScanNet, Veo3 achieves an AbsRel of 0.075 and $\delta_1$ of 0.952, a substantial improvement over prior frozen-feature baselines.
• Camera Pose Estimation: End-point error is competitive (1.10 vs. 1.08 for DINOv2), with optimal extraction points consistent across tasks.

Figure 5: Downstream performance on geometry tasks confirms universal optimal extraction zone identified via probes.

Qualitative Decoding

Preview decoders at intermediate layers and noise levels demonstrate that even simple readout heads recover salient scene structure from diffusion features.

Figure 6: Attention readout head enables temporally coherent video reconstruction from latent diffusion representations.

Figure 7: Probe performance versus network depth and noise level confirms optimal region for feature extraction.

Practical and Theoretical Implications

The paper establishes that highly scaled video diffusion models, trained for generation, also naturally encode general-purpose spatiotemporal representations suitable for a broad array of perception tasks, without explicit contrastive or MAE objectives. This contradicts earlier findings, which suggested diffusion models were primarily geometric encoders and lacked high-level semantics [Velez_2025_ICCV]. Gen4U demonstrates that the latest generation of diffusion models exhibit emergent semantic alignment, suggesting that generative pretraining at scale may obviate the need for modality-specific pretraining pipelines and fine-tuning.

Practically, this enables shared models for both generation and understanding, drastically reducing system complexity and inference cost. Architectures can be frozen, preserving generative capacity, while lightweight adapters provide competitive discriminative performance. Theoretical implications include the confirmation that generative modeling can implicitly support world-model learning and multi-modal alignment, inviting further exploration of representation learning driven by generative objectives.

Future Directions

Areas for development include deeper characterization of the bimodal representation geometry observed in Veo3, expansion to fully open-weight models for reproducibility, and pursuit of state-of-the-art results in broader video perception tasks. The emergence of semantic bottlenecks and the dynamic evolution of features across the generative manifold warrant further study, particularly as model scaling continues.

Conclusion

Gen4U substantiates the claim that diffusion-based generative models, when sufficiently scaled, encode structured representations that unify video generation and understanding in a single system. Through detailed probing and systematic evaluation, the framework achieves strong results across semantic and geometric tasks, supporting efficient, single-pass feature extraction without sacrificing generative ability. The findings challenge established dichotomies between generative and discriminative paradigms, suggesting new directions for unified multimodal foundation models (2607.06856).