RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

Abstract: LLMs are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.

• The paper demonstrates that frozen multimodal language models condition evolving noisy RAE latents to significantly enhance denoising performance.
• It decouples conditional encoding from the denoising backbone via MLP projection and token-aligned AdaLN, achieving superior prompt alignment.
• Ablations reveal that leveraging multimodal perception pretraining as a robust prior outperforms traditional joint-training approaches at equal compute.

RepFusion: Exploiting Multimodal Priors for Denoising in Representation Space

Motivation and Conceptual Framework

"RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space" (2606.14700) elucidates a paradigm shift in text-to-image (T2I) generative pipelines. Conventionally, large pretrained LLMs function solely as static text encoders; denoising in visual representation space is performed downstream by separately trained generative backbones (e.g., diffusion transformers). This division persists despite the evolution of latent spaces from VAE latents to semantically richer RAEs, which are intrinsically compatible with pretrained multimodal LLM (MLLM) priors.

The paper postulates that pretrained MLLMs, especially those with perception pretraining, are not limited to text encoding—they can act as effective noisy representation encoders. By feeding evolving noisy RAE latents (alongside text) into an off-the-shelf MLLM via a simple MLP projection, RepFusion leverages these outputs for conditioning a diffusion transformer (DiT). Crucially, the MLLM backbone remains frozen, enabling retention of rich multimodal priors.

Figure 1: Left: RAEs enable semantically rich latent spaces conducive to MLLM denoising; Right: Parameter efficiency—allocating capacity to conditional encoders (MLLMs) yields superior results over monolithic denoisers under equal inference budgets.

Architectural Design

The RepFusion framework decouples conditional encoding from the denoising backbone. A frozen MLLM processes the text prompt and projected noisy RAE latents using an MLP projector. Its hidden states—output as token-wise multimodal embeddings—condition each DiT block via AdaLN modulation, exploiting the natural token alignment between RAE and MLLM outputs. Only the projector and DiT are trainable; the pretrained MLLM is preserved.

Figure 2: Architecture—frozen MLLM encodes text/noisy RAE latents; outputs are injected into DiT via token-aligned AdaLN modulation.

This approach fundamentally alters capacity allocation. RepFusion achieves superior denoising performance to baselines that spend comparable or greater parameter budgets exclusively on denoising transformers (e.g., Transfusion, TextEmbed), demonstrating the efficacy of leveraging pretrained conditional encoder priors.

Ablations and Empirical Analysis

Noisy Representation Conditioning

RepFusion reveals that conditioning the MLLM on evolving noisy RAE latents is pivotal; learnable query-based approaches (e.g., MetaQuery, BLIP-3o) that omit explicit noisy representation input are unable to benefit from repeated MLLM recomputation at inference, resulting in stagnant performance despite matched training/inference compute.

Multimodal Perception Pretraining

The paper presents robust ablations showing that perception pretraining is a transferable prior for diffusion in RAE space. Substituting language-only LLMs with perception-pretrained MLLMs consistently improves performance, even when the backbone is frozen. Notably, joint fine-tuning of perception-pretrained MLLMs with generative objectives leads to performance degradation relative to freezing, contradicting the common practice of joint training.

Figure 3: Multimodal perception pretraining in the LLM backbone dramatically boosts denoising performance in RAE space.

Stepwise Improvement over Baselines

The progression from static text embedding to RepFusion is dissected: feeding noisy VAE latents into the LLM gives moderate improvement; switching to RAE latents provides a substantial boost due to their semantic richness and compatibility; multimodal perception pretraining and freezing the MLLM backbone yield optimal results.

Figure 4: Ablation trajectory from TextEmbed and Transfusion baselines to RepFusion, showing incremental GenEval score improvements.

Generation Quality and Prompt Alignment

Leveraging only ~30M image-caption pairs, RepFusion achieves competitive prompt alignment on multiple benchmarks, including GenEval, GenEval++, GenEval2, and DPG-Bench. SFT fine-tuning delivers state-of-the-art alignment, and the inherent strengths persist across evaluation protocols designed to reduce synthetic data bias.

Figure 5: RepFusion generates high-fidelity samples with accurate prompt adherence, even for complex or compositional instructions.

Furthermore, reasoning-based generation assessment on the WISE benchmark demonstrates that RepFusion, with a frozen MLLM backbone, matches or outperforms state-of-the-art multimodal generative models in world-knowledge and reasoning tasks.

Scaling and Interface Analysis

RepFusion identifies dual scaling axes: MLLM conditioning and DiT denoising. Empirical analysis evidences that scaling either component improves performance, with DiT scaling generally favored under iso-FLOPs settings. Nonetheless, RepFusion outperforms static text embedding baselines that exhaustively scale the denoising transformer, validating the core thesis.

Figure 6: Performance rises with increases in MLLM and DiT model size, with clearer trends on GenEval and GenEval++.

The AdaLN-Single interface, exploiting token alignment, efficiently injects MLLM hidden states into DiT, outperforming cross-attention mechanisms in both quality and parameter efficiency.

Practical and Theoretical Implications

RepFusion introduces a practical recipe for architectural design: allocate substantial model capacity to pretrained multimodal conditional encoders, preserve their perceptual priors during training, and feed input-dependent (noisy) visual representations during denoising. This shifts the paradigm from static text encoding to dynamic multimodal conditioning, offering improvements in prompt alignment, compositional reasoning, and inference efficiency.

Theoretically, the results underscore the importance of representation space compatibility and prior preservation—multimodal perception pretraining is a transferable, robust prior for generative modeling. The findings will likely influence multimodal generative architectures, prompting further exploration of conditional encoder roles, encoder-decoder parameter allocations, and test-time compute allocation.

Conclusion

RepFusion demonstrates that pretrained MLLMs, when exposed to evolving noisy representations, provide robust and efficient priors for denoising in semantically structured latent spaces. Frozen conditional encoders with perception pretraining outperform conventional approaches that jointly optimize generative and conditional components, particularly in the context of prompt adherence and compositional reasoning. The work sets forth new directions in multimodal generative modeling by elevating the role of conditional encoders and emphasizing the preservation and exploitation of pretrained multimodal priors.