---
title: 'Kandinsky 5.0: Open-Source Generative Models'
url: https://www.emergentmind.com/topics/kandinsky-5-0
type: topic
---

# Kandinsky 5.0: Open-Source Generative Models

Kandinsky 5.0 is a family of large-scale, open-source foundation models for high-resolution image and video generation, comprising state-of-the-art text-to-image, in-context image editing, and text-to-video/image-to-video models. The framework features three principal model line-ups—Image Lite, Video Lite, and Video Pro—spanning a parameter regime from 2 billion to 19 billion. Kandinsky 5.0 leverages a unified Cross-Attention Diffusion Transformer (CrossDiT) backbone, an extensive, highly curated multi-modal dataset pipeline, innovative training techniques including flow matching, advanced supervised and reinforcement learning-based post-training, and extensive optimizations for scalability and throughput. All code, checkpoints, and recipes are freely available under an open-source license, targeting researchers and practitioners seeking extensible, high-quality generative capabilities [2511.14993].

## 1. Model Line-Ups and Core Architecture

Kandinsky 5.0 introduces three main model variants, each optimized for specific generative domains:

| Model Variant          | Parameter Count | Primary Tasks                                               | Max Resolution   | Distinctive Features             |
|-----------------------|-----------------|-------------------------------------------------------------|------------------|----------------------------------|
| Image Lite            | 6B              | Text-to-image, in-context image editing                     | 1408 px          | High resolution, image editing   |
| Video Lite            | 2B              | Text-to-video, image-to-video (≤10s, ≤768 px)               | 768 px           | Fast, lightweight, video focus   |
| Video Pro             | 19B             | High-quality text-to-video and image-to-video (≤10s, HD)    | 1408 px          | Superior video generation        |

All models employ the same architectural backbone: a latent diffusion model whose core is a transformer-based U-Net—CrossDiT. Key CrossDiT modules include interleaved self-attention, cross-attention (with text), MLP blocks with GeLU activations, residual connections after each sub-block, and adaptive normalization layers. Dual text encoders are integrated: Qwen2.5-VL (7B) for dense text representations (dimension 3584, context 256), and CLIP ViT-L/14 (dimension 768, context 77) for additional adaptive normalization. Images are encoded with FLUX.1-dev VAE, while videos use HunyuanVideo VAE for temporal consistency.

Variant-specific architectural hyperparameters are detailed as follows:

| Model        | #CrossDiT Blocks | #LTF Blocks | Linear Dim | Model Emb | Time Emb |
|--------------|------------------|-------------|------------|-----------|----------|
| Image Lite   | 50               | 2           | 10,240     | 2560      | 512      |
| Video Lite   | 32               | 2           | 7168       | 1792      | 512      |
| Video Pro    | 60               | 4           | 16,384     | 4096      | 1024     |

Notable architectural innovations unique to Kandinsky 5.0 include the Linguistic Token Refiner (LTF) for denoising text embeddings pre-fusion, and Neighborhood Adaptive Block-Level Attention (NABLA), a dynamic, block-sparse attention mechanism optimized for video.

## 2. Data Curation Lifecycle

Kandinsky 5.0’s data pipeline is characterized by massive scale, heterogeneity, and rigorous filtering to support robust multi-modal generation.

- **Kandinsky T2I (text-to-image):** 500 million images (LAION/COYO/web, min side ≥256 px).
- **Kandinsky T2V (text-to-video):** 250 million video scenes, 2–60 seconds, various aspect ratios.
- **Kandinsky I2I (image editing instruction):** ~150 million image pairs with edit descriptions.
- **Kandinsky RCC (Russian Cultural Code):** 229k videos, 768k images, manually curated with bilingual captions.
- **Supervised Fine-Tuning (SFT) datasets:** v1 (strict): 2,833 video/45k image; v2 (relaxed): 12,461 video/153k image.

Processing pipelines are domain-specific: images are filtered for resolution, deduplicated via perceptual pHash, cleansed of watermarks (ResNeXt101/YOLO), scored for quality (TOPIQ/Q-Align), text presence (CRAFT), and complexity (SAM 2/Sobel). Captioning uses InternVL2-26B, InternLM3-8B, and Qwen2.5VL-32B, with post-processing for text cleanliness. Video datasets undergo scene segmentation (PySceneDetect), deduplication, multi-stage technical and aesthetic quality assessment, object/scene tagging, synthetic captioning (Tarsier2-7B), and cluster-based sampling (InternVideo2-1B embeddings, 10k-cluster k-means). Instruction datasets apply similarity-based deduplication (CLIP, DINO, face), geometric verification (LoFTR + RANSAC), domain-specific exclusion heuristics, and instruction generation via fine-tuned GLM 4.5.

SFT data are domain- and subdomain-clustered with Qwen2.5-VL-Instruct-32B and CLIP, supporting granular supervised adaptation.

## 3. Multi-Stage Training Methodology

Kandinsky 5.0 employs a multi-regime, multi-stage training pipeline:

- **Pre-training Regimes:** Four settings—text-to-image (T2I), instruct editing, text-to-video (T2V), image-to-video (I2V)—with diffusion in latent space trained via a flow matching objective:

  $$
  \mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{t,\,x_0,\,\varepsilon}\Bigl\|\,v_\theta(x_t, t)\;-\;\frac{x_{t+1}-x_t}{\Delta t}\Bigr\|^2,
  $$
  where $x_t$ are latents, $v_\theta$ is the velocity predictor, and $\Delta t$ is time discretization.

- **Fine-Tuning Regimes:**
    - *Image SFT:* 153k images (EN/RU captions), 9 domains/2–9 subdomains. Finetune per subdomain. “Model-soup” combines checkpoints weighted by $\sqrt{\text{subdomain size}}$.
    - *Video SFT:* 2.8k videos + 45k images per domain, two approaches—standard fine-tuning and model-soup averaging (better stability/quality).

- **Distillation:**  
    - *CFG Distillation*: Reduces from 100 to 50 NFEs, guidance scale $s=5$, teacher trajectory regression.
    - *Consistency/Adversarial Distillation*: TSCD (trajectory-segmented consistency) for students with as few as 16 NFEs; adversarial hinge loss post-training (RMSprop, $\mathrm{lr}_G$ = $10^{-6}$, $\mathrm{lr}_D$ = $10^{-4}$).

- **RL-based Post-training (Image Lite):**  
    - Reward model: Qwen2.5-VL-7B, outputting “Yes” probability for (gen/real, prompt) tuples.
    - DRaFT-K fine-tuning: loss combines reward-based loss and KL ($\beta_{\mathrm{KL}}=2\cdot 10^{-2}$), backpropagating through the last $K=10$ steps.

## 4. Training and Inference Optimization Strategies

Kandinsky 5.0 incorporates pragmatic and analytical training and inference optimizations for both throughput and resource efficiency.

- **Training Optimizations:**
    - Pre-encoded VAE latents are packed in .tar archives, streamed via a 100 Gbps link, dynamically batched by aspect ratio and time budget.
    - Distributed training uses PyTorch FSDP/HSDP with sequence parallelism; text encoder on 32 GPUs, transformer on 64, leveraging NVLink interconnect and async checkpoints.
    - Activation checkpointing and host offloading reduce peak activation memory by 40%.
    - Analytical models provide a priori estimates of training step time and memory:

      $$
      \text{Step} = \frac{d}{d_0} \times  \frac{1}{\,  d_0 + 14\frac{S}{S_0} + 6\frac{d}{d_0}\;  \times L\,B}
      $$
      $$
      \mathrm{Mem} = \frac{12\,L\,(9d_t d + 8d^2 + 2d_f d)}{N} + \max\Bigl(\frac{4\,L\,(9d_t d + 8d^2 + 2d_f d)}{N},\,2S(Ld\,o + 18d +2d_f)\Bigr)
      $$

- **Inference Optimizations:**
    - VAE encoder tiling with torch.compile yields $2.5\times$ speedup.
    - CrossDiT: torch.compile, fused kernels, MagCache (+46% throughput), Flash/Sage attention (≤5s clips), and NABLA for long/HD video.
    - NABLA: blockwise Q/K pooling (N=64), CDF sparsity per attention head, union with sliding-tile patterns and fractal reordering; enables $2.7\times$ speedup at 90% sparsity, negligible perceptual loss.

Empirical performance on a single 80GB H100 GPU:

| Model             | Frames | Resolution    | NFEs | Time (s) | Mem (GB) |
|-------------------|--------|--------------|------|----------|----------|
| Video Lite 5s     | 121    | 512×768      | 100  | 139      | 21       |
| Video Lite Flash  | 121    | 512×768      | 16   | 35       | 21       |
| Video Pro 10s     | 241    | 768×1280     | 100  | 3218*    | 68       |
| Video Pro Flash   | 241    | 768×1280     | 16   | 576*     | 68       |
| Image Lite        | 1      | 1024×1024    | 100  | 13       | 17       |

*With activation offloading.

## 5. Empirical Performance and Evaluation

Evaluation employs both human side-by-side (SBS) assessments and quantitative metrics:

- **Human SBS:** More than 20 annotators (5-way overlap) assessed prompt following (entity/action count, property/placement), and visual quality (composition, lighting, artifact rate, realism, motion coherence) on Elementary.center.
- **Comparative Results:**
    - *Video Lite vs. Sora (OpenAI)* on MovieGen: exceeds Sora in motion, artifact reduction, overall quality in ≥60% of 65k judgments; prompt following is parity.
    - *Video Lite vs. Wan 2.2 5B/14B*: outperforms on visual quality and motion; Wan leads on prompt granularity.
    - *Video Lite vs. Kandinsky 4.1*: 59% motion/visual preference, prompt adherence parity.
    - *Video Pro vs. Veo 3/Fast*: Veo 3 leads in prompt following; K5.0 Pro in visual quality/motion coherence.
- **Quantitative Metrics:**  
    - FVD, VBench, CLIP-Score: K5.0 Pro claims state-of-the-art on VBench and FVD on MovieGen.
    - Distillation: NFE reduction from 100 to 16 demonstrates negligible degradation in FVD/CLIP-Score.
- **Generation Efficiency:**  
    - Video Lite Flash generates a 5s clip in ~35s on H100; Video Pro Flash (10s, 768×1280) in ~576s (with offloading).

## 6. Open-Source Availability and Model Extensibility

All code, training checkpoints, and pipeline integrations are freely accessible:

- **Repositories:** GitHub (https://github.com/kandinskylab/kandinsky-5), HuggingFace (https://huggingface.co/kandinskylab).
- **License:** MIT.
- **Diffusers Integration:** Direct compatibility with the diffusers Python package:

  ```python
  from diffusers import KandinskyV5Img, KandinskyV5Video
  model = KandinskyV5Img.from_pretrained("kandinskylab/kandinsky-5-image-lite")
  pipe = model.pipeline()
  image = pipe("A golden retriever puppy in a meadow at sunrise", guidance_scale=7.5).images[0]
  ```

- **Customization and Adaptation:** Recipes exist for domain-specific SFT (e.g., medical, cartoon, architectural), tuning NABLA sparsity threshold for compute-quality trade-off, and extension to multi-modal tasks (captioning, audio generation). The suite supports “foundation” model construction by weight-averaging across line-ups.

- **Prospective Directions:** Planned updates include longer context (>1024 tokens) text encoders for improved prompt alignment, unification toward a cross-modal T2I/T2V/I2I/I2V/V2A backbone, consumer-level real-time inference via further distillation/quantization, and curriculum-based pretraining to improve rare category generalization and mitigate dataset biases.

Kandinsky 5.0 represents an integrative advance in large-scale generative modeling, combining flexible transformer-diffusion architectures, robust multi-stage data pipelines, sparse attention for video, and advanced fine-tuning/distillation strategies as an open foundation for future research and application [2511.14993].

Source: https://www.emergentmind.com/topics/kandinsky-5-0