RepFusion: Dual Diffusion Methods

• RepFusion is a dual-approach framework that includes a text-to-image generation system which repurposes a multimodal LLM to condition a diffusion transformer on evolving noisy RAE tokens.
• In its recognition formulation, RepFusion transfers rich diffusion-model features to a lightweight student network using dynamic timestep selection optimized by reinforcement learning.
• Both methods leverage structured diffusion-era representations to drive improvements in generation quality and recognition tasks like classification, segmentation, and landmark detection.

RepFusion is the name of two distinct methods in recent arXiv literature. In a 2026 text-to-image formulation, RepFusion is a generation system that combines representation autoencoders (RAEs), large multimodal LLMs (MLLMs), and a diffusion transformer (DiT) so that denoising is conditioned on evolving noisy representation tokens rather than on a static text embedding (Pan et al., 12 Jun 2026). In a 2023 recognition formulation, RepFusion is a knowledge transfer paradigm that extracts diffusion-model activations at different timesteps and distills them into a student network, with the timestep selected dynamically by reinforcement learning (Yang et al., 2023). The shared term therefore denotes a family resemblance around diffusion-era representations, but not a single canonical architecture.

1. Terminological scope and disambiguation

The term “RepFusion” is not unique to one research program. It has been used for a text-to-image generation system and, separately, for a representation distillation framework for recognition tasks.

Usage	Domain	Defining mechanism
RepFusion (Pan et al., 12 Jun 2026)	Text-to-image generation	MLLM conditioning on noisy RAE representations inside the denoising loop
RepFusion (Yang et al., 2023)	Recognition and transfer learning	Distillation of time-indexed diffusion features into a student network

This naming overlap matters because the two methods address different problems, operate on different objects, and optimize different objectives. The 2026 system works in an RAE latent space $\mathcal{Z}$ and treats multimodal priors as a denoising resource. The 2023 system treats a pretrained diffusion probabilistic model (DPM) as a source of teacher features and uses those features for downstream classification, semantic segmentation, and landmark detection. A common misconception is therefore to identify “RepFusion” with only one of these methods.

2. RepFusion in text-to-image generation: motivation and architecture

The 2026 RepFusion begins from a criticism of the standard diffusion-based text-to-image pipeline: “encode text once $\rightarrow$ denoise latents repeatedly.” In that regime, the text encoder runs once to produce a static conditioning embedding, while the diffusion model spends its capacity learning to denoise VAE latents. The paper argues that VAE latents are optimized for reconstruction fidelity and heavy compression rather than semantics, so they are neither linguistically meaningful nor directly readable by an LLM. RAEs change this regime by replacing the VAE bottleneck with a frozen pretrained visual encoder such as CLIP-L/14 or DINO plus a high-capacity decoder, producing a latent space of semantically structured tokens (Pan et al., 12 Jun 2026).

RepFusion’s key architectural move is to repurpose an MLLM that can already ingest clean CLIP-style visual tokens, and extend that mechanism to noisy RAE representations. The system has three module families. First, a frozen RAE encoder $E_{\text{RAE}}$ maps an image to $N$ representation tokens in $\mathcal{Z}$, and a paired high-capacity decoder reconstructs pixels from clean representations. Second, a frozen 7B-parameter causal LLM $E_{\text{LLM}}$ is paired with a small trainable MLP projector $P_\psi$ that maps each noisy representation token into the LLM input embedding space. Third, a standard DiT denoiser operates directly in $\mathcal{Z}$, with its residual blocks modulated by the LLM’s final hidden states for the $N$ visual tokens via AdaLN-Single.

At timestep $t$, the clean representation $\rightarrow$0 is noised to obtain $\rightarrow$1. A timestep embedding $\rightarrow$2 is added to $\rightarrow$3, the MLP projects each token, and the projected visual tokens are appended to the tokenized prompt:

$\rightarrow$4

The LLM processes this sequence and emits hidden states, after which the last $\rightarrow$5 states are extracted as token-aligned conditioning signals for the DiT. The design avoids pooling and cross-attention because the RAE produces exactly $\rightarrow$6 tokens and the LLM consumes exactly $\rightarrow$7 visual tokens beyond the text prompt (Pan et al., 12 Jun 2026).

3. Conditioning on evolving noisy representations

The mathematical formulation of the 2026 RepFusion is a standard variance-preserving diffusion process in the RAE latent space $\rightarrow$8. The forward process is

$\rightarrow$9

with the equivalent sampling form

$E_{\text{RAE}}$0

The denoiser predicts the added noise through

$E_{\text{RAE}}$1

and training minimizes the diffusion MSE in epsilon-prediction form:

$E_{\text{RAE}}$2

The central mechanism is repeated LLM conditioning. At each sampling step,

$E_{\text{RAE}}$3

so the conditioning tokens evolve with the denoising trajectory rather than remaining fixed. The projector is a two-layer perceptron

$E_{\text{RAE}}$4

applied identically to each noisy token after adding the time embedding. Within the DiT, token-wise scale and shift parameters are computed from $E_{\text{RAE}}$5 and used in AdaLN:

$E_{\text{RAE}}$6

This conditioning strategy is contrasted with static text embeddings or learned queries, which produce a constant $E_{\text{RAE}}$7 and cannot adapt to the current denoising state. Empirically, per-step recomputation increases GenEval from 0.54 for learned queries to 0.70 (Pan et al., 12 Jun 2026). The result is presented not merely as a different conditioner, but as a shift in where test-time compute is spent: repeated multimodal inference becomes part of denoising itself.

4. Training, inference, and empirical performance in text-to-image generation

Training for the 2026 system is performed on 128 x H200 GPUs with global batch size 2048. Pretraining uses the BLIP-3o 31 M captioned corpus for 10 epochs and 160k steps at learning rate $E_{\text{RAE}}$8 with AdamW, $E_{\text{RAE}}$9, $N$0, weight decay 0.1, and cosine decay with 10k warmup. A final supervised fine-tuning pass uses a 200k synthetic image-caption set at $N$1 for 64 epochs (Pan et al., 12 Jun 2026).

Inference follows ancestral sampling: draw $N$2 and iteratively apply the reverse update while recomputing $N$3 at every step by rerunning the MLP and the LLM on the current $N$4. The paper emphasizes that this incurs additional compute. One reported comparison moves from 113T to 552T FLOPs when matching a MetaQuery baseline’s per-step LLM calls, and this is associated with a GenEval gain from 0.55 to 0.70.

The reported benchmark results place RepFusion above several baselines under controlled comparisons.

Setting	System	Result
GenEval, no SFT	RepFusion with RAE decoder	0.73
GenEval, no SFT	RepFusion with diffusion decoder	0.78
GenEval, no SFT	Transfusion	0.63
GenEval, no SFT	static TextEmbed	0.47–0.57
GenEval, after SFT	RepFusion with RAE decoder	0.85
GenEval, after SFT	RepFusion with diffusion decoder	0.87

After SFT, the paper reports that RepFusion sets a new state of the art on GenEval, GenEval++, GenEval2 with $N$5, and DPG-Bench with 85.11. The ablations isolate two factors. First, noisy representation inputs are important; replacing them with learned queries substantially degrades performance. Second, multimodal perception pretraining matters; replacing the perception-pretrained MLLM with a language-only LLM backbone drops GenEval by 0.06–0.08 across both Transfusion-RAE and RepFusion, and jointly fine-tuning the MLLM backbone hurts when it is already multimodally pretrained. Scaling studies further show that, under a fixed sampling budget of about 540T FLOPs, larger within-family DiT allocation improves performance, yet RepFusion remains stronger than a TextEmbed baseline that devotes nearly all compute to its denoiser (Pan et al., 12 Jun 2026).

5. RepFusion in recognition: diffusion features as teachers

The 2023 RepFusion addresses a different problem: transferring representational structure from a pretrained diffusion model to a lightweight student network for downstream recognition. The point of departure is that a DPM is trained as a time-conditioned denoiser,

$N$6

and that its intermediate activations, especially in the mid-UNet block, encode rich semantic structure. The paper states that minimizing the DDPM loss

$N$7

decomposes into a representation learning term and a spectral regularization term controlled by the diffusion scale $N$8. Empirically, features at moderate timesteps exhibit higher effective rank and better class separation (Yang et al., 2023).

RepFusion therefore extracts teacher features from a pretrained diffusion denoiser $N$9 at selected timesteps. For an input image $\mathcal{Z}$0 and timestep $\mathcal{Z}$1,

$\mathcal{Z}$2

A student encoder $\mathcal{Z}$3 produces a feature $\mathcal{Z}$4, and the goal is to distill the teacher feature into the student. The paper defines a generic knowledge transfer loss with hint loss, attention transfer, and pairwise relational KD terms, while noting that in practice a single hint loss is sufficient.

A distinctive component is timestep selection by reinforcement learning. A small policy network $\mathcal{Z}$5 chooses the timestep as a per-sample action. The optimal time is defined through task performance of a light decoder $\mathcal{Z}$6 on teacher features,

$\mathcal{Z}$7

Because exhaustive search is impractical, the policy is trained by REINFORCE, with state equal to the raw image, action equal to timestep selection, and reward given by the negative task loss. The objective includes an entropy term with weight $\mathcal{Z}$8 (Yang et al., 2023).

6. Architectures, hyperparameters, and benchmarks for recognition

The 2023 RepFusion uses a DDPM/ADM UNet teacher with $\mathcal{Z}$9 steps and the $E_{\text{LLM}}$0-schedule of Ho et al. 2020. Student encoders include ResNet-18, ResNet-50, WRN-28-2, WRN-28-10, and MobileNetV2 for classification; BiSeNet with ResNet-18 or ResNet-50 backbones for segmentation; and HRNet-w18 or ResNet-50 for keypoint detection. The policy network contains 3 convolutional layers and 3 fully connected layers, with timesteps discretized into $E_{\text{LLM}}$1 bins. Optimization details are task-specific: distillation uses SGD with learning rate 0.1, momentum 0.9, weight decay $E_{\text{LLM}}$2, 200 epochs, batch size 128, and step decay at epochs 100 and 150; segmentation fine-tuning uses SGD with learning rate $E_{\text{LLM}}$3 for 160k iterations, batch size 16, crop size 448, and evaluation at 512; keypoint detection uses Adam with learning rate $E_{\text{LLM}}$4 for 60 epochs and batch size 128 (Yang et al., 2023).

The reported gains span multiple recognition tasks.

Task	Baseline	With RepFusion
CelebAMask-HQ segmentation, ResNet-18	mIoU 61.22	mIoU 67.37
CelebAMask-HQ segmentation, ResNet-50	mIoU 65.81	mIoU 70.63
WFLW landmark detection, HRNet-w18	NME 4.33	NME 4.23
WFLW landmark detection, ResNet-50	NME 4.58	NME 4.47

For semantic segmentation on CelebAMask-HQ with 19 classes, RepFusion raises ResNet-18 from mIoU 61.22 to 67.37, aAcc from 94.65 to 94.75, and mAcc from 70.29 to 76.37, outperforming MoCov2 at 66.38, SwAV at 67.14, and DeepCluster-v2 at 66.92. With ResNet-50, the baseline of 65.81 rises to 70.63, exceeding SwAV at 70.41. For landmark detection on WFLW with 98 points, HRNet-w18 improves from 4.33 to 4.23 NME, including pose 7.81 to 7.42, occlusion 5.35 to 5.10, and illumination 4.42 to 4.18, while ResNet-50 improves from 4.58 to 4.47. For image classification, CIFAR-10 reports a RepFusion gain of +0.42, with the best attention-transfer gain of +0.62 at approximately $E_{\text{LLM}}$5; Tiny-ImageNet reports +1.83 to about 62.2%; and MobileNetV2 reports an average gain of +0.42. The paper characterizes the overall outcome as state-of-the-art or competitive performance across classification, segmentation, and keypoint detection (Yang et al., 2023).

7. Relation between the two meanings of RepFusion

The two methods share a name but solve different optimization problems. The 2026 RepFusion is generative and denoising-centered: it operates in a semantically structured representation space $E_{\text{LLM}}$6, uses a frozen MLLM plus MLP projector to encode noisy RAE tokens, and conditions a DiT through token-aligned AdaLN (Pan et al., 12 Jun 2026). The 2023 RepFusion is transfer-oriented and recognition-centered: it extracts mid-block diffusion representations at selected timesteps and distills them into a student network, with timestep choice handled by an RL policy (Yang et al., 2023).

A plausible commonality is that both methods treat diffusion-era internal representations as primary computational objects rather than auxiliary by-products. In the 2026 system, noisy latent representations are presented to an MLLM so that multimodal priors participate directly in denoising. In the 2023 system, timestep-indexed diffusion features are treated as teacher signals whose usefulness depends on where they lie along the noise trajectory. This suggests a broader shift from viewing diffusion models only as samplers toward viewing them as structured representation processors.

Another misconception is that both methods are variants of distillation. That is accurate only for the 2023 formulation. The 2026 formulation does not distill a diffusion teacher into a student; instead, it reallocates model capacity under an iso-FLOPs perspective by freezing an MLLM and repeatedly invoking it on evolving noisy latents. Conversely, the 2023 formulation does not introduce an MLLM, RAE, or text-to-image generation objective. The overlap is therefore conceptual rather than architectural: both works argue that useful information is distributed across diffusion trajectories and can be exploited more directly than in conventional static-conditioning pipelines.