HunyuanImage 3.0: Multimodal Autoregressive Model

• HunyuanImage 3.0 is a unified multimodal autoregressive model that integrates text and image processing through a Mixture-of-Experts framework with over 80B parameters.
• It employs Chain-of-Thought reasoning and generalized causal attention to manage heterogeneous sequence data for robust text-to-image synthesis.
• The model’s staged training regime, open-source release, and strong evaluation metrics highlight its potential to advance scalable multimodal research.

HunyuanImage 3.0 is a native multimodal autoregressive model that unifies multimodal understanding and generation for text and image modalities. Developed upon a Mixture-of-Experts (MoE) LLM approach, it incorporates advanced architectural innovations—such as Chain-of-Thought (CoT) reasoning and generalized handling of heterogenous sequence data—and is engineered for scalable, efficient training and inference. With over 80 billion total parameters (13 billion activated per token), HunyuanImage 3.0 is presented as the largest open-source image generative model to date. All open source assets, including code and pre-trained weights, are available via its public repository, with documented evaluations showing parity or superiority relative to leading models. HunyuanImage 3.0 builds directly on the foundations laid by Hunyuan-DiT (Li et al., 14 May 2024), which pioneered robust bilingual multimodal text-to-image synthesis and multi-turn interactive capabilities.

1. Architectural Innovations

HunyuanImage 3.0’s core innovation is its unified autoregressive framework for both understanding and generation in text and image domains. The backbone extends Hunyuan-A13B—a decoder-only LLM with a MoE configuration, comprising 64 experts, 8 of which are dynamically activated per token.

Multimodal input sequences are encoded using:

• A pre-trained vision encoder for image understanding.
• A Variational Autoencoder (VAE) projecting pixels to a 32-dimensional latent space, with a downsampling factor of 16.
• Dedicated projector modules (timestep-modulated residual block for VAE features; two-layer MLP for vision features) map latent and visual features into the LLM’s word embedding space.

Visual and textual tokens use a generalized Rotary Position Embedding (RoPE): standard 1D RoPE for text, and a generalized 2D RoPE for images, with the embedding for position $(x, y)$ given by $$[\cos(x\theta_0), \cos(y\theta_1), \ldots, \sin(x\theta_0), \sin(y\theta_1), \ldots]$$, preserving spatial encoding and ensuring backward compatibility with the text-centric format. The attention mechanism is realized as "Generalized Causal Attention": text tokens follow conventional autoregressive causal masking, whereas image tokens attend to prior tokens and within their own segment. For batches generating multiple images, modified attention masks containing “holes” prevent information leakage across image segments.

Analysis shows MoE experts specialize by modality across layers, optimizing text-image fusion and reducing computation by activating only a subset of the MoE at each inference step (13B out of 80B parameters per token).

2. Chain-of-Thought Reasoning Mechanism

HunyuanImage 3.0 employs a native Chain-of-Thought (CoT) approach throughout training and inference. CoT enables multi-step reasoning for image synthesis: the model first interprets the prompt, conducts internal iterative reasoning or refinements, and finally generates the image.

Training uses both text-to-text and text-to-text-to-image reasoning data, explicitly enabling the model to sequence intermediate states of thought before visual output. This process embeds logical and semantic progression within the generation pipeline, supporting complex prompt interpretation and adaptive synthesis. A plausible implication is that such reasoning may increase robustness in structured or multi-turn dialog scenarios (cf. Hunyuan-DiT (Li et al., 14 May 2024), which demonstrated similar interactive capabilities).

3. Training Regime and Optimization

Training is progressively staged:

• Stage I: Low resolution (256px VAE), backbone transformer trained on multimodal and unimodal tasks.
• Stage II: Vision Transformer (ViT) fine-tuning (backbone frozen), focused on enhancing visual representations.
• Stage III: Joint training on interleaved text-image data and higher-resolution images (512px), enabling features such as image editing.
• Stage IV: Training on high-resolution images (minimum 1024px short edge) incorporating reasoning data to support CoT.

Post-training includes:

• Supervised Fine-Tuning (SFT): On curated, high-quality human-annotated sets.
• Direct Preference Optimization (DPO): Mitigates structural distortions.
• MixGRPO: Online RL optimizing for aesthetics, realism, and alignment.
• SRPO: Gradient-guided, single-step latent denoising.
• Reward Distribution Alignment (ReDA): Regularizes output distributions toward high-reward samples.

During inference, the MoE configuration ensures efficient activation (8 of 64 experts per token). Attention masks use the canonical generalized causal structure, with specific adjustments for multi-image scenarios. The VAE encoder progresses in image resolution per training stage while the ViT remains at 512px.

4. Data Curation and Multimodal Pipeline

Performance and generalizability rely on meticulous data curation: image-text pairs are mined from a heterogeneous blend of purchased, open, and partner sources. Data interpretation includes scoring for quality, aesthetics, and safety (e.g., filtering for violence or indecency), followed by hierarchical tiering (copper, silver, gold) to optimize foundation and generative model components, a strategy previously established in Hunyuan-DiT.

Annotation and structural caption refinement use MLLM-based correction, tag injection, and world knowledge enhancements, ensuring semantic depth and robustness in both Chinese and English. This suggests ongoing model evolution is underpinned by these pipeline optimizations.

5. Evaluation Metrics and Results

HunyuanImage 3.0 is evaluated by both automatic and human protocols:

• Structured Semantic Alignment Evaluation (SSAE): Utilizes LLMs and MLLMs to extract 12 semantic fields from prompts (e.g., objects, scene, style, composition). Image outputs are scored for field accuracy, Mean Image Accuracy, and Global Accuracy.
• Good/Same/Bad (GSB): 1,000 prompts, 100+ professional raters conduct pairwise image comparisons.

Table: Key Model Comparison Metrics

Model	Relative Win Rate (%)	Evaluation Protocol
HunyuanImage 3.0	+14.10 vs v2.1	GSB (pairwise, 1k prompts)
Seedream 4.0	Lower	GSB
Nano Banana	Lower	GSB
GPT-Image	Lower	GSB

Qualitative and quantitative analyses indicate win rates comparable or superior to leading models in both text-image alignment and visual quality, demonstrating parity with closed-source benchmarks. Enhanced subject clarity and aesthetics are also noted.

6. Open Source Release and Community Impact

HunyuanImage 3.0 is fully open-sourced, including both codebase and trained weights, with public availability at the specified repository. This transparency is positioned to foster community-driven innovation, reproducibility, and collaborative development.

This release follows the trajectory begun by Hunyuan-DiT (Li et al., 14 May 2024) in supporting scalable multimodal research, now extended to unified autoregressive multimodal foundation modeling.

7. Technical Specifications and Implementation Details

Model backbone: MoE-decoder (Hunyuan-A13B), 80B parameters, 13B activated per token (8/64 experts). Position encoding: generalized 2D RoPE for images, standard 1D RoPE for text, mathematically formulated as:

• $[\cos(n\theta_0), \ldots, \sin(n\theta_0), \ldots]$ for text (position $n$)
• $$[\cos(x\theta_0), \cos(y\theta_1), \ldots, \sin(x\theta_0), \sin(y\theta_1), \ldots]$$ for images (position $(x, y)$)

Attention: Generalized causal, with specialized masks for multi-image tasks to prevent context contamination.

Training pipeline: Staged progressive resolution ramp-up for VAE (256px → 1024px); ViT resolution fixed at 512px. Post-training applies SFT, preference optimization (DPO, MixGRPO), denoising (SRPO), and reward alignment (ReDA).

A plausible implication is that this architectural and training strategy enables HunyuanImage 3.0 to efficiently scale model capacity and expressive power without prohibitive computational overhead.

• (Li et al., 14 May 2024) "Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding"
• (Cao et al., 28 Sep 2025) "HunyuanImage 3.0 Technical Report"