---
title: CrossDiT Diffusion Transformer
url: https://www.emergentmind.com/topics/crossdit-diffusion-transformer
type: topic
---

# CrossDiT Diffusion Transformer

The CrossDiT (Cross-Attention Diffusion Transformer) is the architectural foundation of the Kandinsky 5.0 model family for high-fidelity image and video generation. Engineered as a unified latent-diffusion transformer backbone, CrossDiT is designed to efficiently scale across parameter budgets, tasks, and modalities—including text-to-image (T2I), text-to-video (T2V), and instructive image editing scenarios. It is trained under a flow-matching diffusion paradigm and incorporates recipe-driven optimizations for both throughput and generation quality, exemplifying a modern approach to multi-modal foundation models [2511.14993].

## 1. Model Architecture and Backbone

CrossDiT is based on a stack of modular blocks, each incorporating three principal components: self-attention, cross-attention (conditionally on language embeddings), and a feed-forward network (MLP), each surrounded by pre-normalization and residual connections. All variants use a VAE encoder/decoder—FLUX.1-dev for images and HunyuanVideo VAE for videos—to map raw pixels to a compact latent representation, which serves as input for the transformer-based denoising process.

The text-conditioning pathway uses the Qwen2.5-VL text encoder (7B parameters, outputting 3584-dimensional embeddings), refined through a lightweight Linguistic Token Refiner (LTF) module. For long sequences, especially in video, CrossDiT incorporates Neighborhood-Adaptive Block-level (NABLA) sparse attention to accelerate computation while preserving fidelity.

The architecture is parameterized according to deployment regime:

| Model      | Blocks | LTF blocks | Hidden dim ($d$) | Time dim | FF dim |
|------------|:------:|:----------:|:----------------:|:--------:|:------:|
| Image Lite |   50   |     2      |     10,240       |   512    | 2560   |
| Video Lite |   32   |     2      |      7168        |   512    | 1792   |
| Video Pro  |   60   |     4      |     16,384       |  1024    | 4096   |

Key computational frameworks:

- **Diffusion loss (flow-matching):**

  $$
  \mathcal{L}_\mathrm{diff}= \mathbb{E}_{t,x_0,\epsilon}\left\| \epsilon - \epsilon_\theta\bigl(x_t, t, \mathrm{text}\bigr) \right\|^2,\quad x_t = x_0 + \sigma(t)\,\epsilon,\;\epsilon\sim\mathcal{N}(0, I)
  $$

- **Transformer attention (single head):**

  $$
  \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left( \frac{QK^\top}{\sqrt{d}} \right) V
  $$

- **Flow-matching ODE divergence (RL stage):**

  $$
  \mathrm{KL}(p_{RL}\!\parallel p_{SFT}) = \sum_{t \in \mathcal{T}} \| v_{RL}(x_t, t) - v_{SFT}(x_t, t) \|^2
  $$

## 2. Data Pipeline and Curation

Training the CrossDiT-centric Kandinsky 5.0 models relies on extensive multi-modal curation workflows:

- **Text-to-Image**: Approximately 500 million image-caption pairs are sourced from LAION, COYO, and similar large-scale datasets. Filtering includes minimum resolution (≥256px), perceptual-hash deduplication, watermark detection (ResNeXt 101, YOLO), technical and aesthetic scoring (TOPIQ, Q-Align), text-region exclusion (CRAFT), complexity analysis (SAM 2 and Sobel), and object/scene annotation (YOLOv8, CLIP). English captions are synthesized with InternVL2-26B and refined by InternLM3-8B; Russian with Qwen2.5-VL.

- **Text-to-Video**: Around 250 million video scenes undergo shot segmentation (PySceneDetect), with similar filtering as T2I, supplemented by MS-SSIM motion metrics, watermark checking, DOVER & Q-Align for quality, camera/object dynamics via VideoMAE, and scene clustering using InternVideo2 embeddings with $k$-means.

- **Image Editing (I2I)**: 150 million image pairs filtered by CLIP/DINO similarity (>0.8), RANSAC geometrics, DINO-aligned crop scoring, and other heuristics. Edits are captioned by fine-tuned GLM-4.5; a supervised subset (153k examples) is curated by Q-Align and human annotation.

Supervised fine-tuning data include ~153k image examples (dual-language captions), 2.8k video scenes and 45k images, all grouped into nine domains according to VLM classification or CLIP-linear $k$-means.

## 3. Multi-Stage Training and Optimization

CrossDiT-based models follow a staged training pipeline:

- **Pre-training** uses AdamW, progressive resolution (image: LR→MR→HR; video: 1s→5s→10s), and unconditional loss injection (10%). Batch sizes (Image Lite: 8k→2k; Video Pro: 16k→665) and LR schedules are tuned per regime. EMA decay is set to 0.9999. NABLA is applied for larger samples.
  
- **Supervised Fine-Tuning (SFT)** leverages "model soups"—domain- or subdomain-specific fine-tuning at low learning rates, with checkpoint weight-averaging either equally or $\propto\sqrt{\text{size}}$. Video SFT proceeds analogously.

- **Distillation**, specific to Video Lite/Pro Flash models, involves CFG-guidance distillation (reducing needed function evaluations from 50 to 16 as per recent methods) and subsequent Hinge GAN post-training on re-noised frames using a Logit-Normal schedule.

- **RL-based Fine-Tuning (Image Lite only)** employs a Qwen2.5-VL-7B reward model trained on human preference pairs and applies Direct Reward Fine-Tuning with an RL loss, regularized by KL divergence versus SFT with $\beta_{KL}=2\times10^{-2}$.

## 4. Inference, Acceleration, and System Integration

Inference and system integration for CrossDiT models feature multiple efficiency innovations:

- **VAE Inference**: Tiling and torch.compile yield 2.5× speedup and improved tile-border continuity.
- **Attention Scaling**: For short sequences, Flash/Sage attention is used; for higher resolutions or longer videos, NABLA achieves ≈90% sparsity and 2.7× speedup.
- **MagCache**: Diffusion-step caching accelerates generation by ≈46%.
- **Text Encoder Quantization**: INT4 quantization minimizes memory overhead.
- **Distributed Training**: F/HSDP + Sequence Parallel sharding over 64 GPUs, with activation checkpointing and offloading in the RL stage.
- **Dynamic Batching**: Batches are grouped by aspect ratio and frame count $\sum_k t_k \approx t_{max}$ to maximize GPU efficiency.

Performance metrics (H100, single GPU):

| Model                 | Frames | Resolution | NFEs | Time (s) | Memory (GB) |
|-----------------------|:------:|:----------:|:----:|:--------:|:-----------:|
| Video Lite 5s         |  121   | 512×768    | 100  |   139    |    21       |
| Video Lite 5s Flash   |  121   | 512×768    | 16   |   35     |    21       |
| Video Pro 10s         |  241   | 512×768    | 100  |  1158    |    51       |
| Video Pro 10s Flash   |  241   | 512×768    | 16   |  242     |    51       |
| Image Lite            |    1   | 1024×1024  | 100  |   13     |    17       |

## 5. Empirical Evaluation and Comparison

Automated metrics and human evaluation demonstrate state-of-the-art performance:

- **COCO-30k FID (Image Lite)**: Outperforms Stable Diffusion 2.1 and DALL·E 2.
- **Video Metrics (Video Lite)**: FVD/CLIP/VBench parity or improvement over Wan 2.2 5B and Sora.
- **Side-by-Side Human Study (MovieGen, 1,002 prompts, 5× overlap):**
  - Video Lite vs Sora: 62% object/action fidelity preference, 59% motion/visual quality.
  - Video Lite vs Wan 2.2 5B/A14B: higher visual/motion scores, ~10–15 point drop in prompt adherence.
  - Video Lite vs K4.1: motion (+59%), 27% fewer artifacts.
  - Video Pro vs Veo 3/Fast: outperforms in quality/dynamics, <15% gap in prompt following.
  - Flash-distilled models show <10% drop in visual/motion criteria for substantial speedup.
  - Image Lite exceeds FLUX.1 and Qwen-Image in quality (+0.08) and prompt following (+0.05).
  - Instructional editing matches or exceeds FLUX/Kontext for instruction and aesthetics.

## 6. Applications and Deployment Recommendations

CrossDiT models enable high-resolution T2I (up to 1408px) for commercial, social, and product content; I2I for inpainting, style transfer, and sketch-to-photo translation; and T2V/I2V (5s/10s, up to 1408px, 24fps) for storyboarding, marketing, and virtual sets.

Deployment best practices include:

- Domain-specific SFT soups for style/content adaptation.
- Text encoder quantization (INT4) for inference efficiency.
- Flash/NABLA attention for latency-critical use.
- Flash variants (Video Lite/Pro) for ultra-fast sampling.
- Prompt engineering following standardized templates:
  - T2I: “[Subject] + [Style/Lighting] + [Details]”
  - T2V: “[Subject] + [Action] + [Environment] + [Cinematic params]”

Example usage (Python, diffusers):

```python
from diffusers import DiffusionPipeline, KandinskyImagePipeline, KandinskyVideoPipeline

# Text-to-Image
pipe_img = KandinskyImagePipeline.from_pretrained(
    "kandinskylab/kandinsky-5-image-lite",
    torch_dtype=torch.float16
).to("cuda")
image = pipe_img("A photorealistic portrait of a fox in a forest")[0]

# Text-to-Video (Lite 10s)
pipe_vid = KandinskyVideoPipeline.from_pretrained(
    "kandinskylab/kandinsky-5-video-lite",
    torch_dtype=torch.float16
).to("cuda")
video = pipe_vid("A sleek red car racing down a neon city street at night")[0]
video[0].save("out.mp4", fps=24)

# Image-to-Video
input_frame = load_image("street.png")
i2v = KandinskyVideoPipeline.from_pretrained(
    "kandinskylab/kandinsky-5-video-lite",
    torch_dtype=torch.float16
).to("cuda")
clip = i2v(prompt="Camera pans upward", init_frame=input_frame)[0]
clip.save("pan.mp4", fps=24)
```

## 7. Context and Impact within Foundation Models

CrossDiT, as instantiated in Kandinsky 5.0, exemplifies a modular, scalable approach to generative multi-modal learning, unifying high-fidelity image and video synthesis in a single transformer design leveraging latent diffusion, advanced attention mechanisms, and large-scale curation pipelines. Its open-source (MIT) release and comprehensive checkpoints facilitate downstream adaptation and benchmarking. The backbone integrates recent innovations in diffusion acceleration, clustering-based data domainization, and reward-model-driven fine-tuning, contributing to the accessible deployment of foundation models for both research and applied creative domains [2511.14993].

Source: https://www.emergentmind.com/topics/crossdit-diffusion-transformer