T2I-Diff: Advances in Text-to-Image Diffusion

• T2I-Diff is a family of text-to-image diffusion systems that use iterative denoising and cross-attention to map text prompts to images through latent or pixel representations.
• It features diverse architectures including UNet-based latent diffusion, transformer (DiT) models, and cascaded pixel-space systems with specialized inversion and control mechanisms.
• Recent innovations address multi-modal conditioning, evaluation, and reward modeling to enhance prompt adherence, diversity, and compositional accuracy in generated images.

T2I-Diff most commonly denotes text-to-image diffusion: a family of generative systems that map a text prompt to an image by iteratively denoising a latent or pixel-space representation under textual conditioning. In current usage, the term covers latent diffusion models such as Stable Diffusion, diffusion transformers and MMDiT-style rectified-flow systems such as SD 3.5 and FLUX, and a growing ecosystem of control, evaluation, and fine-tuning methods built around them (Mou et al., 2023, Li et al., 14 Feb 2026). A separate 2025 work uses the same title, “T2I-Diff,” for an fMRI generation framework based on time-frequency images and classifier-free diffusion (Tew et al., 25 Sep 2025).

1. Core formulation and architectural families

A canonical T2I-Diff pipeline encodes textual input, injects that conditioning into a denoiser, and learns to reverse a forward noising process. In latent diffusion, an autoencoder maps an image $\mathbf{X}_0$ to a latent $\mathbf{Z}_0$, noise is added according to

$$\mathbf{Z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{Z}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},$$

and the model is trained with the standard noise-prediction objective

$$\mathcal{L} = \mathbb{E}_{\mathbf{Z}_{t}, \mathbf{C}, \boldsymbol{\epsilon}, t} \big[ \, \|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\mathbf{Z}_t, \mathbf{C}) \|_2^2 \big].$$

Text is typically injected through cross-attention,

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\Big(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\Big)\cdot \mathbf{V},$$

with image-derived queries and text-derived keys and values (Mou et al., 2023).

More recent T2I-Diff systems shift from UNet-centered latent diffusion to transformer backbones such as DiT and MMDiT, often under rectified flow or flow-matching objectives. In the MMDiT picture emphasized by Diff-Aid, text tokens and image tokens interact through joint self-attention,

$$\mathbf{z}_{t}^{l+1} = \text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d}}\right)\mathbf{V} + \mathbf{z}_{t}^{l},$$

and the central failure mode is not the absence of conditioning but the mis-calibration of text–image interaction across blocks, timesteps, and tokens (Li et al., 14 Feb 2026).

A second architectural branch keeps diffusion in pixel space rather than a VAE latent. DeepFloyd-IF, for example, is a cascaded three-stage system with a low-resolution text-to-image stage followed by super-resolution stages. This avoids the lossy first compression stage of latent diffusion, but it complicates inversion and editing because the super-resolution diffusion models are not compatible with standard DDIM inversion; IterInv addresses this by using NTI in the low-resolution stage and iterative variance updates in later stages (Tang et al., 2023).

2. Representations, scaling, and datasets

One major design axis in T2I-Diff is the latent representation itself. Classical latent diffusion uses a VAE latent optimized for reconstruction. SVG-T2I instead performs diffusion directly in the feature space of a visual foundation model, specifically DINOv3, replacing the usual VAE latent with DINOv3 feature maps. Its autoencoder uses a frozen DINOv3-ViT-S/16+ encoder, the main generator is a 2.6B-parameter Unified Next-DiT, and the overall system reaches 0.75 on GenEval and 85.78 on DPG-Bench, showing that VFM features can support large-scale high-resolution T2I synthesis (Shi et al., 12 Dec 2025). The paper also makes explicit the trade-off between semantically structured but resolution-sensitive VFM latents and more resolution-invariant but less semantic VAE latents. This suggests that latent choice is not merely an implementation detail but a central axis of T2I-Diff design.

A second axis is data. Fine-T2I was introduced specifically to close the open-data gap in T2I fine-tuning: the final release contains over 6 million text-image pairs, roughly 2 TB on disk, with a synthetic set of 6,145,693 pairs and a curated real-image set of 168,424 pairs. It spans 10 task combinations, 32 prompt categories, 11 visual styles, and 5 prompt templates, and more than 95% of initial candidates are filtered out before release (Ma et al., 10 Feb 2026). Fine-tuning on Fine-T2I improves both generation quality and instruction adherence across pretrained diffusion and autoregressive models; on GenEval, SD-XL rises from 0.55 to 0.61, while LlamaGen rises from 0.32 to 0.41 (Ma et al., 10 Feb 2026).

The interaction between representation and data curation is increasingly central. SVG-T2I shows that a DINO-space generator can be competitive with VAE-based systems at scale, while Fine-T2I shows that fine-tuning-level quality at multi-million-sample scale materially changes instruction following and visual quality. Together they indicate that T2I-Diff progress is no longer reducible to backbone size alone (Shi et al., 12 Dec 2025, Ma et al., 10 Feb 2026).

3. Conditioning and controllability

Prompt-only conditioning is often insufficient when precise structure, attributes, or multimodal inputs are required. The contemporary T2I-Diff literature therefore adds lightweight control modules, adaptive conditioning layers, and explicit geometry-aware interfaces.

Method	Mechanism	Scope
T2I-Adapter	Additive multi-scale adapters on frozen SD encoder features	Sketch, depth, segmentation, keypose, color
Diff-Aid	Per-token adaptive modulation across blocks and timesteps	Prompt adherence and downstream plug-ins
AI-T2I	Aggregating-and-isolating cross-attention losses	Precise token-region alignment
POCI-Diff	3D layout guidance with BLD and IP-Adapter	Multi-object scene control and editing
CoT-Diff	MLLM-guided 3D layout planning during denoising	Inline reasoning for spatial control
X2I	Attention distillation from MLLMs into DiT	Multimodal-to-image conditioning

T2I-Adapter established a particularly influential recipe: freeze the base Stable Diffusion model, encode an external control signal with a lightweight CNN, and inject the resulting multi-scale features additively into the UNet encoder. Separate adapters can be trained for sketches, segmentation maps, keypoints, depth maps, and spatial color palettes, and multiple adapters can be composed at inference by linear combination (Mou et al., 2023). Diff-Aid operates at a different point in the stack: rather than adding new control features, it directly modulates the backbone’s use of text during denoising by learning adaptive per-token weights $\alpha_t^l$ and replacing $\mathbf{c}_t^l$ with

$$\widetilde{\mathbf{c}_t^l} = \mathbf{c}_t^l + \mathbf{c}_t^l \odot \alpha_t^l.$$

On SD 3.5 and FLUX, this inference-time plug-and-play module improves prompt adherence, visual quality, and human preference, including a GenEval increase from 0.72 to 0.77 for SD 3.5 and from 0.68 to 0.70 for FLUX (Li et al., 14 Feb 2026).

A parallel thread focuses on attention geometry itself. AI-T2I argues that current alignment failures arise from both inter-subject-token overlap and intra-subject-token scattering in cross-attention maps, and introduces an aggregation loss to consolidate scattered activations and an isolation loss to separate different tokens spatially (Cao et al., 25 May 2026). This remains training-free and inference-time, making it compatible with Stable Diffusion, SDXL, SD 3.5, FLUX.1-dev, HunyuanDiT, and Qwen-Image (Cao et al., 25 May 2026).

For explicit scene structure, POCI-Diff and CoT-Diff move T2I-Diff toward 3D-aware generation. POCI-Diff binds per-object prompts to 3D bounding boxes through Blended Latent Diffusion, uses depth-conditioned ControlNet for geometry, and adds IP-Adapter-based reference conditioning for identity-consistent editing (Rigo et al., 20 Jan 2026). CoT-Diff goes further by entangling diffusion with an MLLM that performs 3D scene planning and iterative layout refinement at each denoising step; on 3D Scene benchmarks it reports a 34.7% state-of-the-art improvement in complex scene spatial accuracy (Liu et al., 6 Jul 2025).

X2I extends the conditioning interface from text to multimodal understanding. It freezes a Flux.1-style Diffusion Transformer, replaces the original text encoders with an MLLM plus an AlignNet bridge, and distills the teacher’s attention behavior so that the DiT can condition on multilingual text, screenshots, images, videos, and audio with less than 1% degradation relative to the teacher while gaining new multimodal capabilities (Ma et al., 8 Mar 2025).

4. Evaluation, reward modeling, and model selection

Benchmarking T2I-Diff has shifted from generic fidelity scores toward fine-grained instruction following. TIIF-Bench was designed precisely for this: it contains 5000 prompts organized across multiple dimensions and three difficulty levels, supplies both short and long versions of each prompt with identical core semantics, introduces text rendering and style control as explicit evaluation axes, and adds 100 high-quality designer level prompts for realistic professional scenarios (Wei et al., 2 Jun 2025). Its computable framework decomposes prompts into fine-grained yes/no questions rather than scoring a full prompt monolithically.

Other benchmarks emphasize complementary aspects. GenEval focuses on object presence, counting, colors, and positions; T2I-CompBench++ decomposes performance into color, texture, shape, non-spatial relations, 2D spatial, 3D spatial, numeracy, and complex prompts; DPG-Bench measures broader semantic alignment (Shi et al., 12 Dec 2025, Shahabadi et al., 12 Dec 2025). On these suites, diffusion models remain uneven. The comparative study “Infinity and Beyond” finds that diffusion systems such as SDXL and PixArt-$\alpha$ show persistent weaknesses in attribute-sensitive and spatial tasks, while Flux-Dev and Flux-Schnell improve numeracy and multi-object behavior but still trail the strongest VAR models in overall compositional alignment (Shahabadi et al., 12 Dec 2025).

Reward modeling is increasingly part of the T2I-Diff stack rather than a separate evaluation layer. AutoRubric-T2I replaces opaque scalar Bradley–Terry reward models with a rubric-learning framework that synthesizes explicit criteria from only 256 preference pairs, scores prompt–image pairs with a VLM judge, and refines the rubric set by $\mathbf{Z}_0$0-regularized logistic regression (Kao et al., 17 May 2026). Its final scalar reward is

$\mathbf{Z}_0$1

where each $\mathbf{Z}_0$2 is the VLM-estimated probability that the image satisfies rubric $\mathbf{Z}_0$3. On MMRB2, AutoRubric-T2I with Gemini-3-Flash on PickScore reaches 71.4% overall, above 59.4% for HPSv3 and 59.8% for UnifiedReward, and it improves downstream diffusion RL in Flow-GRPO on TIIF and UniGenBench++ (Kao et al., 17 May 2026).

The surrounding infrastructure has also become an object of study. DiffAgent treats the T2I ecosystem itself as a tool-use problem: given a prompt, an LLM agent selects a suitable model and parameter configuration from a large API space. With DABench, it trains on 50,482 instruction–API pairs, drawn from a broader Civitai-scale environment of 74,492 distinct models, and reports an average API-generation time of 4.81 seconds (Zhao et al., 2024). This reframes part of T2I-Diff practice as model routing rather than model design.

5. Failure modes, stereotypes, and overcorrection

A persistent misconception is that visually strong T2I-Diff models necessarily follow prompts accurately. Diff-Aid makes the counterpoint explicit: SD 3.5 and FLUX can produce visually impressive but prompt-inaccurate images because text–image interaction is insufficient or mis-calibrated across blocks, timesteps, and tokens. Typical failures include missing attributes such as rendering a “yellow stop sign” as a red sign, missing objects, and incorrect relations or counts (Li et al., 14 Feb 2026).

Compositional alignment remains a core bottleneck. The diffusion-versus-VAR comparison in “Infinity and Beyond” shows that even strong diffusion transformers remain weaker on several spatial and attribute-binding tasks than the best autoregressive alternatives, especially for complex prompts and 2D/3D spatial relations (Shahabadi et al., 12 Dec 2025). This does not imply that diffusion is intrinsically incapable, but it does indicate that the denoising objective alone has not solved prompt faithfulness.

Fairness interventions expose a second failure mode: overcorrection. DIVBENCH distinguishes under-diversification, where models reproduce narrow stereotypes, from over-diversification, where demographic attributes are modified even when prompts or contexts constrain them. The benchmark uses 20 unassigned attribute prompts, 20 assigned gender prompts, and 20 assigned race prompts, and evaluates diversification through Precision, Recall, and F1. It finds that many diversification strategies improve recall at the cost of lower precision, while context-aware methods—especially LLM-guided prompt rewriting and context-aware FairDiffusion—better balance diversity with semantic fidelity (Friedrich et al., 2 Jul 2025).

OASIS frames the related but distinct issue of stereotypes in sociological rather than parity-based terms. Its Stereotype Score

$\mathbf{Z}_0$4

measures directional over-representation of stereotypical attributes relative to real-world prevalence, while WAlS measures spectral variance along those attribute directions (Dehdashtian et al., 1 Jan 2025). Using OASIS, the authors conclude that newer T2I models such as FLUX.1 and SDv3 still contain strong stereotypical predispositions, and that stereotype severity worsens for nationalities with lower Internet footprints (Dehdashtian et al., 1 Jan 2025).

6. Extensions beyond standard image synthesis

Although T2I-Diff is primarily an image-synthesis paradigm, its priors and mechanisms have already migrated into adjacent areas. IterInv shows that inversion-based editing can be extended from latent diffusion to pixel-level cascaded models such as DeepFloyd-IF by treating later super-resolution stages with iterative inversion rather than standard DDIM inversion. In the reported experiments, IterInv yields reconstructions that are visually far closer to the original images than DDIM inversion on the same pipeline, and it enables prompt-based editing with DiffEdit-style methods in a pixel-space T2I stack (Tang et al., 2023).

DP-IQA repurposes Stable Diffusion’s denoising prior for blind image quality assessment in the wild. It uses the pre-trained Stable Diffusion U-Net as a multi-level feature extractor, adds a tunable text adapter and an image adapter, and then distills the resulting teacher into an EfficientNet-B7 student. The teacher reaches state-of-the-art performance on several authentic IQA datasets, including 0.907 / 0.879 PLCC/SRCC on CLIVE and 0.951 / 0.942 on KonIQ, demonstrating that T2I diffusion priors can serve as strong perceptual backbones outside generation itself (Fu et al., 2024).

A separate line of work uses the exact title “T2I-Diff” for an fMRI generation framework that converts BOLD signals into complex spectrogram images, applies an EDM-style classifier-free diffusion model in that image space, and reconstructs time-domain signals via inverse Fourier transforms (Tew et al., 25 Sep 2025). In that usage, “T2I” refers to time-to-image, not text-to-image. The reuse of the label does not change the dominant meaning of T2I-Diff in generative modeling, but it does illustrate how diffusion workflows originally developed for image synthesis now function as more general priors for structured signal generation (Tew et al., 25 Sep 2025).

Across these developments, T2I-Diff has evolved from a prompt-conditioned image generator into a broader technical stack: a family of latent and pixel-space generative models; a substrate for plug-and-play control, 3D layout reasoning, and multimodal conditioning; a target of increasingly fine-grained evaluation and reward modeling; and, in some cases, a reusable prior outside image synthesis altogether (Li et al., 14 Feb 2026, Fu et al., 2024).