2000 character limit reached
HunyuanImage 3.0 Technical Report (2509.23951v1)
Published 28 Sep 2025 in cs.CV
Abstract: We present HunyuanImage 3.0, a native multimodal model that unifies multimodal understanding and generation within an autoregressive framework, with its image generation module publicly available. The achievement of HunyuanImage 3.0 relies on several key components, including meticulous data curation, advanced architecture design, a native Chain-of-Thoughts schema, progressive model pre-training, aggressive model post-training, and an efficient infrastructure that enables large-scale training and inference. With these advancements, we successfully trained a Mixture-of-Experts (MoE) model comprising over 80 billion parameters in total, with 13 billion parameters activated per token during inference, making it the largest and most powerful open-source image generative model to date. We conducted extensive experiments and the results of automatic and human evaluation of text-image alignment and visual quality demonstrate that HunyuanImage 3.0 rivals previous state-of-the-art models. By releasing the code and weights of HunyuanImage 3.0, we aim to enable the community to explore new ideas with a state-of-the-art foundation model, fostering a dynamic and vibrant multimodal ecosystem. All open source assets are publicly available at https://github.com/Tencent-Hunyuan/HunyuanImage-3.0
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces HunyuanImage 3.0, a powerful open-source AI model that can both understand images and create new images from text. Think of it as a smart artist and reader in one: it can read a prompt, plan how to draw it, and then paint a detailed, high-quality picture that matches the description. The model is built on a very large “language brain” and adds vision abilities so it can handle pictures and words together.
What were the main goals?
- Build one single model that can do both image understanding and image generation, instead of having separate systems for each.
- Make the model plan its work using a “think first, draw later” process (called Chain-of-Thought), so it follows instructions better and produces more accurate images.
- Train the largest and most capable open-source image generator so researchers and developers can use and improve it.
- Create better ways to prepare data and evaluate how well text and images match.
How did they build and train the model?
The team used a mix of careful data work, a specially designed model, and step-by-step training to teach the model to understand and generate images at a high level.
Collecting and cleaning data
They started with over 10 billion images and kept only the best ones (less than 45%). They removed:
- Low-quality or broken images, duplicates, and images with too much text or watermarks.
- A lot of AI-generated images, which can confuse training.
They also added special sets of images (like posters, diagrams, and stylized art) to make the training more diverse. Besides single pictures, they built a large set of related image pairs and multi-image sequences (including frames from videos) to teach the model about relationships and edits across images.
Writing good captions for the images
To help the model learn, images need solid descriptions. The team built:
- A bilingual (English/Chinese) caption system that describes images at different detail levels, includes style and lighting, and names real-world things (like characters, landmarks, or brands).
- A method to mix and match different caption parts so the model sees many styles of prompts, from short (about 30 words) to very long (up to 1,000 words).
- Two helpers: an OCR agent (reads text inside images) and a named-entity agent (recognizes real people, places, logos). They cross-check these facts with the captions so the training data stays accurate.
- A “difference captioner” that explains how two similar images differ, which helps the model learn image editing instructions.
Teaching the model to “think” before drawing
They trained the model on:
- Text-to-Text (T2T) data: The model practices turning short prompts into clearer, step-by-step plans. This improves logic and instruction-following.
- Text-to-Text-to-Image (T2TI) data: The model learns the full process—read the prompt, reason about it, rewrite a detailed plan, then generate the image. This “think first, draw second” approach makes results more accurate.
The model’s design (explained simply)
HunyuanImage 3.0 is a “native multimodal” model: it handles words and images in one unified system.
- A giant language brain (LLM with Mixture-of-Experts): Imagine a school with 64 specialist teachers (experts). For each word or image chunk, only a few teachers (8) are activated. This makes the model both huge (over 80 billion total parameters) and efficient (it uses about 13 billion per step), so it can be smart without being too slow.
- A vision encoder and a VAE: The vision encoder “reads” images like a careful viewer. The VAE turns images into a smaller, easier-to-handle form (like making a high-quality thumbnail) that’s perfect for image generation.
- A projector: This is a translator that maps vision features into the same space the language brain understands, so words and pictures can “talk” to each other.
- Diffusion inside the LLM: Diffusion is like starting with a noisy canvas and cleaning it step by step until a clear picture appears. Here, the language brain helps guide that clean-up process so the final image matches the text well.
- Smart attention: When writing text, the model only looks backward (like writing a sentence word by word). But when painting, it needs to see the whole canvas to keep shapes and colors consistent. The model combines both styles so words stay logical and images stay coherent.
- Position understanding in 2D: The model uses a 2D sense of “where” for image pieces (like coordinates on a grid) and a 1D sense for text (like positions in a sentence). This helps it place image parts correctly.
- Automatic size and shape: The model can decide a good image size and aspect ratio based on the prompt (for example, “vertical portrait” vs. “wide landscape”), or follow what the user asks.
Training in stages and polishing
They trained the model in four stages, starting with lower-resolution images and simpler tasks, then moving to higher resolution and more complex tasks. They included:
- Language-only training (so it follows instructions well),
- Multimodal understanding (so it can read and discuss images),
- Text-to-image generation,
- Interleaved text-and-image sequences (dialogues with pictures, and editing),
- And finally, reasoning (Chain-of-Thought) for better planning.
After that, they did several rounds of post-training to polish image quality:
- SFT (Supervised Fine-Tuning): Learn from carefully chosen high-quality examples.
- DPO: Learn from comparisons between better and worse images to reduce distortions.
- MixGRPO: A reinforcement learning method that improves how well images match the text and look good (composition, lighting, style).
- SRPO: Another reinforcement method that focuses on realism and fixes common issues like oversaturation or odd lighting.
- ReDA: Aligns the model’s output with the “distribution” of very high-quality images.
What did they find?
- High-quality results: On their new evaluation method (SSAE), which carefully checks how well an image matches many detailed points from a prompt, HunyuanImage 3.0 performs on par with top models across many categories.
- Human preference wins: In a head-to-head human test (GSB: Good/Same/Bad) over 1,000 prompts, HunyuanImage 3.0 beat their previous open-source model (HunyuanImage 2.1) by a strong margin, and also edged out several well-known closed-source models in overall quality.
- Specialist “experts” emerge: Inside the Mixture-of-Experts system, some experts naturally specialize more in text and others in image processing. This suggests the MoE design helps the model handle different types of information more effectively.
Why this is important: It shows that one unified model can “think, read, and draw” well, and that advanced polishing steps make a real difference to image look and prompt accuracy.
Why does it matter?
- Open-source power: They released the code and weights for the image generation part, making cutting-edge image AI more accessible to students, researchers, and developers.
- One model for many tasks: A single system that can understand and generate images makes future tools simpler, more flexible, and more reliable.
- Better planning = better pictures: Teaching the model to plan (Chain-of-Thought) leads to more accurate images that follow complicated prompts, which is useful for education, design, advertising, and more.
- Strong data practices: Their careful data filtering, captioning, and fact-checking pipeline sets a good example for building trustworthy AI systems.
- A foundation for the future: While this release focuses on text-to-image, the same framework can support image editing and image-to-image tasks, which the team plans to release soon. This could lead to creative tools that are easier to use and produce results closer to what people imagine.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper advances multimodal image generation, but several aspects remain missing, uncertain, or unexplored. The following list identifies concrete gaps that future work could address:
- Dataset transparency and reproducibility
- Provide a detailed breakdown of the 5B-image corpus (source domains, geographic/demographic distribution, licensing, proportion of internet vs. proprietary sources) and release a replicable subset or data cards to enable independent verification and bias audits.
- Quantify the false-positive/false-negative rates of all filtering detectors (watermark/logo/text/OCR, collage/border, AIGC) and report how these filters affect downstream model performance and dataset composition (e.g., removal of valuable design/graphic/text-heavy images).
- Evaluate the impact of removing AIGC images on convergence, diversity, and realism; determine optimal synthetic–real mixing ratios rather than blanket exclusion.
- Captioning and grounding pipeline
- Report accuracy, coverage, and failure modes of the OCR Agent and the Named Entity (IP) Agent (e.g., entity disambiguation, multilingual text recognition, brand/artwork detection), including quantitative error analysis and human validation.
- Quantify the benefits and risks of Compositional Caption Synthesis (e.g., whether synthetic recombinations introduce stylistic artifacts or unrealistic co-occurrences) via controlled ablations.
- Extend captioning beyond English/Chinese and evaluate multilingual robustness (script variety, code-switching, low-resource languages) with standardized benchmarks.
- Reasoning data and Chain-of-Thought (CoT)
- Specify the exact sizes, class balance, and annotation protocols for T2T and T2TI corpora; provide inter-annotator agreement and quality-control metrics.
- Include ablations isolating the contribution of CoT fine-tuning to text-to-image quality, alignment, and robustness (short vs. long prompts; compositional instructions; OOD tasks).
- Evaluate whether CoT improves or harms sample diversity, and analyze potential overfitting to long-form, schema-structured prompts.
- Architecture choices and ablations
- Provide empirical ablations validating the claim that a single 16× downsampling VAE outperforms 8× VAE + patchification (effects on fine detail, textures, text rendering, high-frequency content).
- Compare the dual-encoder concatenation approach (VAE + vision encoder) against alternatives (e.g., late fusion, cross-attention conditioning) for both generation and understanding tasks.
- Quantify the benefits and costs of Generalized Causal Attention (memory/compute overhead, stability) relative to full attention and standard causal masks; include task-wise ablations (T2I, MMU, INTL).
- Assess how increasing expert specialization across layers (observed KL divergence trend) affects cross-modal interactions, generalization, and potential modality silos or interference.
- Position embeddings and resolution control
- Provide ablations showing the impact of Generalized 2D RoPE on performance versus 1D RoPE and other 2D schemes (e.g., ALiBi, learned PE), particularly at high resolutions and unusual aspect ratios.
- Evaluate the accuracy of automatic size/ratio token prediction across diverse prompts and conditional images; characterize failure cases and user override behaviors.
- Test aspect ratios and resolutions beyond the 1:4–4:1 and current anchors (e.g., panoramic, ultra-wide, tall banners) and analyze degradation modes.
- Training regime and reproducibility
- Disclose full training hyperparameters (optimizer, learning-rate schedules, batch sizes, gradient clipping, loss weights per task), stage durations/steps, and data mixture proportions to enable faithful reproduction.
- Analyze catastrophic forgetting or instability when freezing/unfreezing components across stages, and report mitigation strategies.
- Clarify compute requirements (GPU types/counts, throughput, wall-clock training time) and provide scaling curves for data and parameters.
- Post-training (SFT, DPO, MixGRPO, SRPO, ReDA)
- Publish formal algorithmic details, pseudocode, and hyperparameters for MixGRPO, SRPO, and ReDA; release code where feasible to permit independent replication.
- Define and validate the reward models (open-source vs. proprietary), including calibration, drift, and potential reward hacking; report sensitivity analyses and generalization to unseen domains.
- Quantify improvements attributable to each post-training stage via controlled ablations and standardized metrics (alignment, realism, artifacts), including statistical significance.
- Evaluation methodology
- Validate SSAE (LLM/MLLM-based scoring) against blinded human studies; report inter-rater reliability, calibration procedures, and error analyses (e.g., attribute binding, spatial relations).
- Release the prompts, seeds, and generated images for SSAE and GSB to enable third-party reproducibility; include statistical tests (confidence intervals, effect sizes) for reported win rates.
- Evaluate multilingual prompts, OCR-heavy instructions, and domain-specific scenarios (e.g., UI design, scientific diagrams) not covered or only lightly represented in current benchmarks.
- Capabilities coverage and release scope
- The public release currently covers only text-to-image; provide timelines and evaluations for image-to-image editing, multi-image conditioning, interleaved dialogue, and difference-instruction editing.
- Assess performance on multimodal understanding tasks (VQA, captioning, retrieval) within the unified framework to substantiate “native multimodal” claims.
- Efficiency and deployment
- Report inference speed, memory footprint, and latency on common hardware; evaluate quantization, distillation, or LoRA-style adapters for edge deployment.
- Analyze MoE routing (top‑k, capacity factor, load balancing, dropped tokens) and its impact on throughput and stability during inference.
- Safety, ethics, and IP
- Conduct red‑teaming and safety evaluations (harmful content, bias, stereotypes, sensitive attributes); describe moderation tools and default safeguards.
- Detail watermarking, provenance, and IP/brand handling (given the IP Agent) to mitigate misuse (deepfakes, trademark/character replication) and comply with legal/ethical standards.
- VAE specifics and reconstruction quality
- Provide VAE architecture details (latent channels, encoder/decoder structure, KL weighting), reconstruction metrics, and their correlation with downstream generative fidelity and text rendering.
- Robustness and failure modes
- Systematically characterize failure cases (e.g., attribute binding errors, compositional reasoning mistakes, text rendering inaccuracies, artifacts at extreme resolutions) and propose targeted datasets or training adjustments to address them.
Practical Applications
Immediate Applications
The items below outline concrete, deployable use cases that can be built today using the released HunyuanImage 3.0 weights/code and the paper’s data and training pipelines.
- Automated multi-ratio creative generation (Advertising/Marketing, Media, Software)
- Use cases: Batch-generate campaign assets across aspect ratios (1:1, 9:16, 16:9, posters, thumbnails) with strong prompt-following and accurate text rendering for ads, social, and banners.
- Tools/workflows: REST/SDK inference API; plug-ins for Photoshop/Figma; DAM/CRM integration that leverages “automatic resolution” tokens for on-the-fly sizing.
- Assumptions/dependencies: Adequate GPU serving; human review for brand/safety; licensing checks for outputs.
- E-commerce product imagery at scale (Retail, Marketplaces)
- Use cases: Generate clean product hero shots, backgrounds, seasonal variants, and A/B test visuals aligned to marketplace image specs using automatic size/ratio selection.
- Tools/workflows: Integration with PIM/CMS; batch generation pipelines for product launches; post-processing QA with OCR to verify in-image text.
- Assumptions/dependencies: Domain prompting/fine-tuning for specific product categories; moderation guardrails.
- Bilingual creative and localization (Media, Education, Government comms)
- Use cases: English/Chinese prompts to create localized posters, event flyers, public notices with legible text in-image; quick adaptations across markets.
- Tools/workflows: Localization pipeline where LLM rewrites/clarifies prompts (CoT), then generates visuals; glossaries for brand terminology.
- Assumptions/dependencies: Text-rendering robustness varies by font/script; human proofreading for critical signage.
- UI and graphic design ideation (Software, Product Design)
- Use cases: Turn natural language briefs into UI/poster/mockup concepts; iterate via “think-then-draw” CoT to refine layout, style, and composition.
- Tools/workflows: Design copilot inside Figma/Sketch; prompt templates for wireframes vs. high-fidelity mockups.
- Assumptions/dependencies: Style alignment to design systems requires prompt engineering or light SFT; human-in-the-loop remains essential.
- Educational illustrations and infographics (Education, Publishing)
- Use cases: Generate explanatory visuals for lectures, textbooks, and paper guides; produce infographic-style images from structured prompts.
- Tools/workflows: LMS add-ons; notebook plugins for teaching; templated prompts for curriculum topics.
- Assumptions/dependencies: Factual accuracy must be reviewed; avoid use where misrepresentation could cause harm.
- Open evaluation and model selection with SSAE (Academia, ML Ops, Policy Audits)
- Use cases: Replace noisy CLIP-only metrics with structured semantic alignment evaluation (SSAE) that better mirrors human judgment for T2I model selection.
- Tools/workflows: Integrate SSAE into CI for generative model releases; leaderboard curation; crowd-in-the-loop verification on critical tasks.
- Assumptions/dependencies: Requires capable MLLM evaluators and compute; prompt taxonomy coverage must match target domain.
- Industrial data curation and content moderation (Platforms, Data Vendors, Trust & Safety)
- Use cases: Deploy the paper’s AIGC detection, watermark/logo/large-text filters, clarity and aesthetics scoring to clean large-scale image corpora or moderate user uploads.
- Tools/workflows: Data ingestion pipelines; deduplication via embedding clustering; continuous detector re-training.
- Assumptions/dependencies: Detector drift over time; need for domain-specific thresholds; privacy/legal compliance.
- High-quality captioning pipelines (Search/Indexing, Accessibility, Data Labeling)
- Use cases: Generate hierarchical bilingual captions; synthesize diverse caption variants; use OCR and Named-Entity (IP) agents with bidirectional verification for fact-grounded metadata.
- Tools/workflows: Alt-text generation for accessibility; dataset enrichment for retrieval; instruction data for downstream VLMs.
- Assumptions/dependencies: OCR/NER accuracy on stylized text; IP/brand compliance policies; agent orchestration costs.
- Post-training recipes for better T2I (Labs, Model Builders)
- Use cases: Apply SFT + DPO + MixGRPO + SRPO + ReDA to improve aesthetics, realism, and text-image alignment in proprietary models.
- Tools/workflows: Reward-model hubs; hybrid ODE–SDE samplers (MixGRPO); differentiable rewards (SRPO) for single-step quality improvements.
- Assumptions/dependencies: Reward model quality; compute budgets; careful reward balancing to avoid regressions.
- Engineering patterns for unified multimodal modeling (ML Systems, OSS Community)
- Use cases: Adopt Generalized Causal Attention, Generalized 2D RoPE, dual-encoder projection, and automatic resolution tokens to build your own native multimodal models.
- Tools/workflows: PyTorch-based reference code; ablations/benchmarks (GSB, SSAE) in CI; MoE instrumentation for expert analysis.
- Assumptions/dependencies: Integration complexity; careful positional alignment in training vs. inference.
- Everyday creative tools and mobile apps (Consumer)
- Use cases: Personal posters, invitations, album art, social content with correct aspect ratios and on-image text.
- Tools/workflows: Mobile app with presets; prompt templates; simple safety filters.
- Assumptions/dependencies: Inference costs and latency; user-friendly safety and rights guidance.
- Platform-level AIGC provenance checks (Policy, Trust & Safety)
- Use cases: Use AIGC detectors from the data pipeline to flag suspected AI images, prioritize reviews, and inform provenance labeling.
- Tools/workflows: Upload-time scanning; risk-based routing to human moderation; audit logs for regulators.
- Assumptions/dependencies: False positives/negatives; continuous updates to counter evasion; coordination with watermarking standards.
Long-Term Applications
These opportunities are enabled by the paper’s architectural choices and datasets but require further research, scaling, or feature releases (e.g., full image-to-image editing module).
- Unified multimodal assistants that “plan then draw” (Productivity, Creative Suites)
- Use cases: Conversational agents that reason about user intent (CoT), propose visual plans, generate drafts, and iterate with edits in a single context.
- Tools/workflows: Chat + canvas interfaces; session memory with interleaved text–image history; safety and style governance.
- Assumptions/dependencies: Full release of understanding + generation + editing in one runtime; robust guardrails.
- High-fidelity image editing and controllable I2I (Design, Photography, Mobile)
- Use cases: Localized edits, style transfer, in/outpainting guided by “difference captions” that describe precise changes.
- Tools/workflows: Layer-aware editing UI; brush + text hybrid control; versioning of edit histories.
- Assumptions/dependencies: Completion of image-to-image training and release; fine-grained conditioning APIs.
- Enterprise “content factory” pipelines (Retail, Media, Gaming)
- Use cases: End-to-end generation of brand-compliant assets across channels and regions with automatic ratio/size routing and approval workflows.
- Tools/workflows: Brand style adapters; preference-optimized RL (e.g., SRPO/ReDA) tuned on brand libraries; traceability dashboards.
- Assumptions/dependencies: Enterprise-grade governance, licensing, and human approvals; domain SFT.
- Scientific and technical visualization (R&D, Education, Healthcare communication)
- Use cases: From structured prompts/specs to accurate diagrams, laboratory setups, and didactic visuals with CoT-backed reasoning traces.
- Tools/workflows: Notebook plug-ins (e.g., Jupyter); prompt schemas for experimental conditions; validation checklists.
- Assumptions/dependencies: Domain adaptation to minimize hallucinations; expert review for safety-critical contexts.
- Accessible UI generation and front-end scaffolding (Software, Accessibility)
- Use cases: Natural language to UI wireframes and visual comps, then handoff to code LLMs for HTML/CSS/React.
- Tools/workflows: Design-to-code bridges; CoT prompt decomposition into components and constraints.
- Assumptions/dependencies: Tight integration with code models; design system alignment and testability.
- Synthetic data generation for model training (Autonomy, Robotics, Healthcare)
- Use cases: Create domain-specific synthetic imagery to augment scarce or sensitive datasets (e.g., rare objects, controlled lighting).
- Tools/workflows: Scenario libraries; distribution matching via ReDA; bias audits with SSAE-style evaluations.
- Assumptions/dependencies: Regulatory acceptance; rigorous bias/safety validation; domain SFT or adapters.
- Text-to-video and storyboard-to-clip extensions (Media, Advertising, Education)
- Use cases: Extend the architecture to temporal generation, using interleaved frames and 2D RoPE generalizations for video.
- Tools/workflows: Shot planning via CoT; video segment mining for training; temporal consistency rewards.
- Assumptions/dependencies: Significant compute; long-context attention scaling; rights management for motion content.
- Standards for auditing and certification (Policy, Industry Consortia)
- Use cases: SSAE-like structured semantic audits become part of compliance and disclosure for generative services.
- Tools/workflows: Public benchmarks with field-level scores; third-party certification; continuous evaluation protocols.
- Assumptions/dependencies: Cross-stakeholder agreement on taxonomies and thresholds; transparent evaluator models.
- Efficient multimodal MoE routing (ML Systems, Edge/On-prem)
- Use cases: Expert specialization by modality to reduce inference cost or meet on-prem constraints while preserving quality.
- Tools/workflows: Dynamic routing policies; expert pruning/merging; telemetry for modal load patterns.
- Assumptions/dependencies: Hardware-aware scheduling; stability under distribution shifts.
- Global multilingual support beyond EN/ZH (Media, Public Sector)
- Use cases: Localized campaigns and public information materials across many languages with reliable on-image text.
- Tools/workflows: Language adapters; font/rendering packs; locale-aware prompt libraries.
- Assumptions/dependencies: Additional multilingual training data; typographic coverage; QA for critical communications.
Glossary
- Activated parameters per token: The number of model parameters that are actually used for processing each token during inference in an MoE model; lower activation can reduce compute while maintaining capacity. "13 billion parameters activated per token during inference"
- Aesthetics model: A learned model that scores images for visual appeal based on attributes like color, lighting, and composition. "Based on this criterion, we build our own aesthetics model."
- AI-generated content (AIGC): Images or media that are synthesized by artificial intelligence models rather than captured from the real world. "AI-generated content (AIGC)."
- Aspect ratio: The proportional relationship between an image’s width and height, often expressed as W:H. "aspect ratio ranging from 1:4 to 4:1."
- Autoregressive next-token prediction: A modeling approach where the model predicts the next token in a sequence based on previously generated tokens, preserving causality. "text tokens are modeled via autoregressive next-token prediction"
- Bidirectional Verification Loop: A process that cross-checks detected entities against generated captions to ensure factual grounding and consistency. "we establish a Bidirectional Verification Loop that cross-references the entities detected by the agents with the generated caption."
- Camera motion classification operator: A tool that labels video segments based on the type or extent of camera movement to filter unsuitable clips. "Camera motion classification operator was subsequently employed to exclude clips exhibiting excessive camera transformation."
- Causal attention: An attention mechanism restricting each token to attend only to past tokens, ensuring autoregressive generation. "Causal attention is a fundamental component in LLMs for autoregressive text generation"
- Chain-of-Thought (CoT): A training and inference approach where the model generates intermediate reasoning steps before producing the final output. "an automated Chain-of-Thought (CoT) reasoning process for image generation"
- Compositional Caption Synthesis: A data augmentation technique that composes captions by sampling and combining structured fields to increase diversity and control. "we introduce Compositional Caption Synthesis, a dynamic data augmentation strategy"
- Conditional image (Cond Image): A previously generated or provided image that conditions subsequent tokens or generations in a sequence. "it is treated as a conditional image (Cond Image) for subsequent tokens in the sequence."
- Decoder-only LLM: A transformer architecture composed solely of decoder blocks, commonly used for generative language modeling. "a decoder-only LLM over 80 billion total parameters."
- Diffusion-based prediction framework: A generative modeling approach where data is produced by iteratively denoising from noise, adapted here for image token prediction. "image tokens are modeled through a diffusion-based prediction framework"
- Diffusion models: Generative models that learn to synthesize data by reversing a noising process through iterative denoising. "particularly diffusion models~\cite{ho2020denoising, song2020denoising, dhariwal2021diffusion, song2020score, rombach2022high, karras2022elucidating, lipman2022flow, liu2022flow, zhang2022unsupervised, zhang2023shiftddpms}"
- DiT-like architectures: Diffusion Transformer architectures that use transformer blocks for diffusion-based image generation. "DiT-like~\cite{dit} architectures~\cite{sd3,hunyuandit,Flux,hunyuanvideo}"
- Direct Preference Optimization (DPO): A post-training method that optimizes the model based on preference pairs, improving outputs aligned with human judgments. "DPO~\cite{wallace2024diffusion} is implemented to effectively address and reduce physical distortions."
- Embedding cluster: Groupings of items (e.g., images) in embedding space used for deduplication or semantic organization. "deduplicated data based on embedding cluster results"
- Flow-based models: Models that learn invertible transformations between simple and complex distributions, enabling exact likelihoods. "extends GRPO to flow-based models"
- Full attention: An attention pattern where every token can attend to all tokens in a segment, useful for capturing global dependencies (e.g., across image patches). "full attention is commonly employed in DiTs for image generation"
- Generalized 2D RoPE: An extension of Rotary Position Embedding to two-dimensional coordinates, allowing spatially-aware image token positioning with backward compatibility to text. "we implement a Generalized 2D RoPE"
- Generalized Causal Attention: A multimodal attention scheme combining causal restrictions for text tokens and global attention for image tokens within segments. "we introduce a Generalized Causal Attention mechanism."
- GRPO: A reinforcement learning method (Guided Reward Policy Optimization) used to optimize generative models based on reward signals. "extends GRPO to flow-based models through a hybrid ODE–SDE sampling strategy."
- Image Difference Captioning: A captioning approach that describes changes between paired images, often simulating editing instructions. "Image Difference Captioning."
- Interleaved text-image modeling (INTL): Joint modeling of sequences that mix text and images, enabling complex multimodal interactions. "including text-to-image generation (T2I), language modeling (LM), multimodal understanding (MMU), interleaved text-image modeling (INTL) and reasoning (CoT)."
- KL divergence: A measure of difference between probability distributions, used here to compare expert activation distributions across modalities. "KL divergence between"
- Latent space: A compressed representation space (often continuous) in which images are encoded by models like VAEs for efficient modeling. "projects raw pixel values into a 32-dimensional latent space"
- LLM: A high-capacity neural model trained on large text corpora to perform diverse language tasks. "a pre-trained Mixture-of-Experts (MoE) LLM"
- MixGRPO: An online reinforcement learning framework extending GRPO with hybrid sampling to improve aesthetic quality, realism, and alignment. "MixGRPO is an efficient online reinforcement learning framework that extends GRPO to flow-based models through a hybrid ODE–SDE sampling strategy."
- Motion blur detection operator: A detector that identifies frames degraded by motion blur to improve dataset quality. "motion blur detection operator"
- Multimodal LLM (MLLM): A LLM that processes and reasons over multiple modalities (e.g., text and images). "advanced LLMs and MLLMs"
- Multimodal understanding (MMU): Tasks where a model interprets and reasons about input across multiple modalities. "multimodal understanding (MMU)"
- Named Entity (IP) Agent: A specialized agent that detects in-image named entities (e.g., characters, landmarks) to provide factual grounding. "a Named Entity (IP) Agent identifies real-world entities."
- OCR Agent: A component that extracts text present in images for use in captioning or grounding. "An OCR Agent extracts in-image text"
- ODE–SDE sampling strategy: A hybrid approach combining deterministic (ODE) and stochastic (SDE) sampling for generative model training. "hybrid ODE–SDE sampling strategy."
- Patchification: Transforming feature maps into patches (tokens) for transformer processing, often reducing spatial resolution. "an additional patchification layer"
- Position embedding: Encodings added to tokens to represent their positions, enabling models to capture order and spatial relationships. "the position embedding is defined as"
- Projector: A module that maps features (e.g., from VAE or vision encoder) into the transformer’s latent space for unified modeling. "We design two distinct projectors modules to align features from the dual image encoders into the transformer's latent space."
- Residual block: A neural network block with skip connections that helps train deep models and stabilize feature transformation. "timestep-modulated residual block"
- Resolution anchor: A target size to which images are resized (preserving aspect ratio), used to control training resolutions. "Resolution anchor denotes that the images are resized to the desired size while keeping the aspect ratio."
- Reward Distribution Alignment (ReDA): A post-training algorithm that aligns the model’s outputs to a high-reward distribution to improve visual quality. "Reward Distribution Alignment (ReDA) method"
- Rotary Position Embedding (RoPE): A position encoding technique that represents positions through rotations in feature space, aiding scalability. "Rotary Position Embedding (RoPE)~\cite{rope}"
- SFT: Supervised fine-tuning on curated datasets to refine model behavior before preference or RL-based post-training. "We first conduct SFT on a meticulously curated dataset of human-annotated examples."
- SRPO: A gradient-guided reinforcement learning strategy that optimizes realism and aesthetics by denoising from a noise prior. "SRPO is a novel gradient-guided online reinforcement training strategy designed to enhance the realism and aesthetic quality of generated images."
- SSAE: Structured Semantic Alignment Evaluation, a benchmark using LLMs/MLLMs to assess fine-grained text-image alignment. "we propose a structured semantic alignment evaluation metric, abbr., SSAE."
- Text-to-Image (T2I): Tasks where a model generates images conditioned on textual prompts. "text-to-image generation (T2I)"
- Text-to-Text (T2T): Reasoning tasks where text prompts are transformed into refined textual outputs to improve instruction following. "Text-to-Text (T2T) reasoning data"
- Text-to-Text-to-Image (T2TI): End-to-end reasoning tasks mapping text prompts to intermediate textual specifications and then to images. "Text-to-Text-to-Image (T2TI) reasoning data"
- Timestep embedding: Encodings of diffusion timesteps injected into sequences to condition the denoising process. "we incorporate timestep embedding into the sequence"
- Transformer backbone: The core transformer network (stack of attention and feed-forward layers) used to process multimodal sequences. "we train the Transformer backbone"
- Variational Autoencoder (VAE): A generative encoder-decoder model that learns a probabilistic latent space for images. "we augment it with a pre-trained vision encoder and a VAE"
- Vision encoder: A neural encoder that extracts visual features from images for understanding and conditioning generation. "a pre-trained vision encoder"
- Vision Transformer (ViT): A transformer-based vision model that processes images as sequences of patches. "vision encoder (ViT)"
Collections
Sign up for free to add this paper to one or more collections.