---
title: 'HunyuanVideo: Scalable Multimodal Video Generation'
url: https://www.emergentmind.com/topics/hunyuanvideo
type: topic
---

# HunyuanVideo: Scalable Multimodal Video Generation

HunyuanVideo is a family of large-scale video models in the Tencent Hunyuan ecosystem spanning text-to-video and image-to-video generation, domain-adapted animation and human animation, video-to-audio synthesis, and structured video comprehension. In the literature surveyed here, the name refers both to an earlier 13B-class generative backbone reused by later adaptation and acceleration studies, and to HunyuanVideo 1.5, an 8.3B-parameter open-source successor implemented as a two-stage system consisting of a unified Diffusion Transformer video generator and a video super-resolution model [2511.18870]. Across these works, HunyuanVideo is characterized by a 3D latent video representation, transformer-based denoising, strong multimodal conditioning, and unusually broad reuse as a substrate for specialization and inference-time optimization [2512.18614].

## 1. Architectural foundations

In PTTA, HunyuanVideo is described as a **large latent diffusion model** trained in a VAE latent space, with a **causal 3D VAE**, a multimodal large language model text encoder, and a **DiT-based** denoising backbone at **13B-parameter-scale**. For a video $\mathbf{v}$ with latent $\mathbf{z}_0 = \text{VAE.encode}(\mathbf{v})$, the forward process is
$$
\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon},
$$
and the conditioning-aware denoiser is trained with the latent-space MSE objective
$$
\mathcal{L}_{\text{LDM}} =
\mathbb{E}_{\mathbf{z}_0, \mathbf{c}, \boldsymbol{\epsilon}, t}
\left[
\left\|
\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\mathbf{z}_t, t, \mathbf{c})
\right\|^2
\right].
$$
PTTA keeps this generative skeleton unchanged and adapts it only through HydraLoRA inserted into the linear layers of the transformer backbone [2512.18614].

Other descendants describe the inherited base somewhat differently. HunyuanVideo-HOMA states that its generator is modified from **HunyuanVideo-T2V**, “including an MMDiT-based video diffusion model and a 3D VAE,” and characterizes the inherited training paradigm as **flow matching** rather than DDPM-style denoising. HunyuanVideo-Avatar similarly adopts **HunyuanVideo-I2V** with an **MM-DiT** backbone and 3D VAE, and FastLightGen treats HunyuanVideo as a large flow-matching DiT video model whose generic objective is written as
$$
\mathop{\arg\min}_{\Theta} \int_0^1 \mathbb{E}\left[\alpha_t\Vert \partial p_t(\mathbf{x}_t)/\partial t - \mathbf{v}_\Theta(\mathbf{x}_t,t)\Vert_2^2\right]dt.
$$
These descriptions indicate that the HunyuanVideo lineage is consistently organized around latent video transformers and multimodal conditioning, even though different papers foreground different training parameterizations and interfaces [2506.08797][2505.20156][2603.01685].

## 2. HunyuanVideo 1.5 as a unified open-source system

HunyuanVideo 1.5 is presented as a compact but high-capacity successor with a **single 8.3B-parameter backbone** and a **two-stage pipeline**: a unified DiT video generator for **T2V**, **I2V**, and **T2I**, followed by a **video super-resolution** model that upscales outputs to **1080p**. The generator operates in a 3D latent space produced by a causal 3D VAE with **$16\times$ spatial compression**, **$4\times$ temporal compression**, and **32 latent channels**. The DiT backbone comprises **54 dual-stream blocks**, **model dimension 2048**, **FFN dimension 8192**, **16 attention heads**, and **head dimension 128** [2511.18870].

A defining systems contribution of HunyuanVideo 1.5 is **Selective and Sliding Tile Attention (SSTA)**, a parameter-free block-sparse attention mechanism designed for long, high-resolution video latents. SSTA computes block-level similarity and redundancy scores,
$$
\mathrm{Score}_i = \lambda \cdot \mathrm{Score}_s - \beta \cdot \mathrm{Score}_r,
$$
selects top-$k$ important blocks, combines these with a local sliding-window mask, and executes the resulting pattern with a custom block-attention kernel. On **720p, 241-frame T2V**, SSTA reduces per-step time from **5.5070 s** to **2.9475 s** without engineering-level tricks; with SageAttention, `torch.compile`, and feature caching, total 50-step runtime falls from **96.78 s** to **58.39 s** [2511.18870].

The conditioning stack is also distinctive. HunyuanVideo 1.5 uses a **dual text encoder** combining **Qwen2.5-VL** for semantic understanding and **Glyph-ByT5** for glyph-aware multilingual text processing, particularly for Chinese. I2V conditioning combines **VAE latent concatenation** of the reference image with the noisy video latent and **SigLIP semantic embedding concatenation** with text tokens. A learnable **type embedding** indicates whether the current task is T2I, T2V, or I2V [2511.18870].

Training is progressive in both modality and scale. The report describes **5B images** at 256p, **1B images** at 512p, then video stages from **256p, 16 fps, 2–10 s** through **720p, 24 fps, 2–10 s**, followed by continuing training for T2V and I2V, supervised fine-tuning, and RLHF. The filtered video corpus contains approximately **800M high-quality video segments** after multi-stage curation from a raw pool of **more than 10 million hours**. This design supports native generation at **480p–720p**, **16–24 fps**, and **2–10 seconds**, while still permitting **720p, 121-frame** generation on a single consumer GPU with **13.6 GB peak memory** via offloading and VAE tiling [2511.18870].

## 3. Task-specific descendants and ecosystem expansion

HunyuanVideo has functioned less as a single model than as a reusable substrate for domain-specific systems. In PTTA, the base HunyuanVideo backbone is frozen and adapted to animation with **HydraLoRA** inserted into all linear layers of the DiT. Training uses **over 12,000** text–animation pairs, **rank = 32**, **alpha = 32**, and a default **$N=4$** HydraLoRA heads corresponding to **male, female, object, background**. On VideoScore metrics, PTTA improves over HunyuanVideo(Base) from **2.823 to 2.895** in VSVQ, from **2.858 to 2.933** in VSDD, and from **2.713 to 3.078** in VSTVA, while maintaining competitive temporal coherence [2512.18614].

HunyuanVideo-HOMA extends the backbone to **generic human-object interaction** generation under weak supervision. Its inputs include a reference human image, reference object image, sparse arm pose, sparse object-center trajectory dots, optional text, and optional audio. It introduces dual-space appearance and motion fusion, a parameter-space **HOI adapter** inserted into even-numbered MMDiT layers, and a facial audio cross-attention adapter. On the self-collected HOI benchmark, HOMA reports **FID 51.60**, **FVD 502.69**, **Object CLIP 90.05**, and **Sync-C 4.33**, and outperforms baselines on both the complex HOI test set and the AnchorCrafter benchmark without per-object finetuning [2506.08797].

HunyuanVideo-Avatar further specializes the family for **audio-driven human animation**, including **multi-character dialogue videos**. It retains the HunyuanVideo-I2V MM-DiT core and adds three modules: a **Character Image Injection Module (CIM)**, an **Audio Emotion Module (AEM)**, and a **Face-Aware Audio Adapter (FAA)**. Training uses approximately **500,000** filtered clips, about **1,250 hours** of video, with a two-stage regimen over **160 GPUs**. On a wild full-body test set, the model attains **IQA 4.66**, **ASE 3.03**, **Sync-C 5.56**, and **FID 49.38**, while a user study reports the best **identity preservation (4.84)** and **lip synchronization (4.65)** among compared systems [2505.20156].

HunyuanVideo-Foley occupies the audio side of the ecosystem. It is a **text–video-to-audio** model that pairs video features from **SigLIP2**, text features from **CLAP**, audio latents from a **DAC-VAE**, and a hybrid **MMDiT + unimodal audio DiT** generator. Its dataset pipeline curates approximately **100k hours** of text–video–audio triplets, and its **REPA** strategy aligns internal audio features to **ATST-Frame** representations via a cosine-similarity loss. On Kling-Audio-Eval it achieves **FD\(_\text{PaNNs}\) 6.07**, **KL 1.89**, **PQ 6.12**, **IB 0.38**, and **DeSync 0.54**, all reported as best among compared baselines in that table [2508.16930].

The name also now covers a comprehension branch. ARC-Hunyuan-Video-7B is presented as the **structured video understanding** counterpart within the broader HunyuanVideo ecosystem rather than a generator. Built on Hunyuan-7B VLM with a ViT visual encoder, Whisper audio encoder, and timestamp-overlaid frame sampling, it supports timestamped captioning, summarization, QA, grounding, and reasoning. It reaches **74.3** accuracy on **ShortVid-Bench**, **54.8 mIoU** on **Charades-STA**, and processes a one-minute video in about **10 seconds** on an **H20 GPU** [2507.20939].

## 4. Inference acceleration and systems research

A large portion of recent efficient video-generation research uses HunyuanVideo as a principal benchmark. This suggests that its long contexts, transformer-heavy compute profile, and open implementations have made it a canonical stress test for inference-time optimization [2505.18875][2605.21042].

| Method | Mechanism | Reported HunyuanVideo result |
|---|---|---|
| AdaSpa [2502.21079] | Dynamic block-sparse attention with online precise search | **1.78×** speedup on 8 s, 720p generation; VBench **80.13** vs **80.10** dense |
| SVG2 [2505.18875] | Semantic-aware permutation sparse attention | **2.30×** FP16 / **2.55×** FP8 speedup on 720p, 33-frame T2V; PSNR **30.452**, VBench **0.852** |
| Fresco [2601.07462] | Progressive spatial resolution with unified noise field | **4.68×** latency and **4.92×** FLOPs reduction on **720×1280**, **125 frames**; Total VBench **80.76** vs **80.12** |
| DPCache [2602.22654] | Path-aware key-timestep selection with cached feature prediction | **4.75×** speedup at **640×480×65**; VBench **80.23** vs **80.93**, memory overhead **+0.36 GB** |
| HyCa [2510.04188] | Dimension-wise hybrid ODE-solver feature caching | **5.56×** FLOPs speedup; VBench **80.25** vs **80.66** |
| Chipmunk [2506.03275] | Dynamic column-sparse deltas over cached activations | **2.16×** alone, **3.72×** with step caching; VBench total **82.94** alone |
| LVSA [2605.31057] | Windowed sparse attention with rotating global anchors | **3.33×** speedup at **193 frames** on HunyuanVideo 1.5; enables **257-frame** generation where dense is OOM |
| DVG [2605.21042] | Content-aware dynamic allocation across time and space | Up to **7.35×** on HunyuanVideo T2V and **18.09×** on distilled HunyuanVideo-1.5 I2V |
| FastLightGen [2603.01685] | Joint step-and-parameter distillation | **4-step**, **70%**-parameter HunyuanVideo-ATI2V with roughly **35.71×** theoretical speedup over 50-step CFG |

These methods target different bottlenecks. Sparse-attention systems such as AdaSpa, SVG2, LVSA, and Chipmunk exploit the fact that HunyuanVideo is strongly attention-bound: AdaSpa estimates roughly **500 PFLOPs** of attention out of **600 PFLOPs** for an **8-second 720p** sample, while SVG2 reports that attention can take **more than 80%** of end-to-end time for a **33-frame** HunyuanVideo run with about **118,800 tokens** per denoising step [2502.21079][2505.18875]. Dynamic-resolution systems such as Fresco and DVG instead reshape the latent grid over denoising stages. Caching methods such as DPCache and HyCa exploit the smoothness of cross-step hidden trajectories, while FastLightGen changes the model itself through pruning and few-step distillation rather than remaining purely training-free [2601.07462][2602.22654][2510.04188][2603.01685].

## 5. Evaluation, deployment characteristics, and practical usage

HunyuanVideo 1.5 is evaluated mainly with human-centric rating and GSB comparisons. In T2V ratings, it reports **61.57** for instruction following, **63.30** for aesthetic quality, **57.35** for visual quality, **79.75** for structural stability, and **57.67** for motion effects. In GSB, HunyuanVideo 1.5 at **720p** records positive win rates over **Wan2.2 (+17.12%)**, **Kling2.1 Master (+12.60%)**, and **Seedance Pro (+11.02%)**, while remaining behind **Veo3 (-10.32%)**. In I2V GSB, it is positive against **Wan2.2 (+12.65%)** and **Kling2.1 Master (+9.72%)**, and slightly negative against **Seedance Pro (-5.77%)** and **Veo3 (-3.61%)** [2511.18870].

A complementary deployment-oriented picture appears in LLMPopcorn, which treats HunyuanVideo as one of three open-source video generators in an LLM-assisted micro-video pipeline. There, HunyuanVideo uses **12,016 MB** of VRAM and **54 seconds per video** on a single **NVIDIA H100 80G GPU**, outperforming CogVideoX-5B in most pairwise popularity comparisons and remaining close to LTX-Video. Under the basic prompt setting, HunyuanVideo wins **56%** of comparisons against CogVideoX-5B on concrete prompts and **51%** against LTX-Video, but under prompt enhancement it falls to **40%** against LTX-Video on concrete prompts, suggesting that it is competitive yet not uniformly the most responsive generator to popularity-oriented prompt engineering [2502.12945].

Beyond generation, the HunyuanVideo-named ecosystem is already used in production comprehension settings. ARC-Hunyuan-Video-7B has been deployed in Tencent products for brief summaries, detailed summaries, and recommendation-oriented “extended browsing words,” with reported retrieval **CTR +5.88%**, landing-page consumption time **+5.11%**, and video floating-layer click CTR **+7.26%** after downstream finetuning [2507.20939]. This does not make ARC-Hunyuan-Video a generator, but it shows that the HunyuanVideo label now denotes a broader multimodal video stack rather than only a denoising backbone.

## 6. Limitations, robustness, and research directions

Several limitations recur across the literature. In animation transfer, base HunyuanVideo is trained primarily on natural videos and exhibits **geometric inconsistencies**, **style mismatch**, and less suitable temporal behavior on anime prompts; PTTA explicitly reports issues such as “misaligned books,” realistic shading where line art is desired, and motion that is not tuned to stylized animation physics [2512.18614]. In weakly conditioned HOI generation, HOMA remains sensitive to the distance between the hand skeleton and object trajectory dot, uses no explicit physics or collision losses, and depends on a weakly labeled dataset of about **140 hours** [2506.08797]. In audio-driven avatar animation, HunyuanVideo-Avatar still relies on **emotion reference images** rather than direct emotion extraction from audio, and a **10-second video at 720×1216 and 50 diffusion steps** is reported to take about **60 minutes** on the authors’ hardware [2505.20156]. HunyuanVideo 1.5 itself, despite strong open-source performance, still trails Veo3 on some T2V human-evaluation axes, especially instruction following and motion effects [2511.18870].

Prompt robustness has also become an explicit research theme. T2VAttack evaluates HunyuanVideo as a black-box victim model and finds it to be the **most robust** among the four tested systems, partly attributing this to an integrated **prompt rewrite mechanism**. Even so, adversarial prompt perturbations remain substantial: on T2VAttackBench, HunyuanVideo’s semantic score falls from **83.0** to **58.8** under substitution-based semantic attack, and its temporal score falls from **72.1** to **36.6** under substitution-based temporal attack; insertion attacks are weaker but still significant [2512.23953].

Future directions are converging around longer horizons, tighter multimodal integration, and more explicit control. PTTA points toward longer narratives and finer style controls for animation; HOMA motivates improved weak controls and stronger human–object disentanglement; HunyuanVideo-Avatar identifies direct audio emotion modeling and real-time avatar generation as open problems; HunyuanVideo-Foley suggests eventual end-to-end text-to-video-plus-audio generation; and long-video acceleration work such as LVSA and DVG points toward content-aware spatiotemporal scheduling for horizons beyond current training lengths [2512.18614][2506.08797][2505.20156][2508.16930][2605.31057][2605.21042]. Taken together, these works indicate that HunyuanVideo has evolved from a single generative backbone into a broader research program around scalable, controllable, and increasingly multimodal video systems.

Source: https://www.emergentmind.com/topics/hunyuanvideo