Representation Conditioning for Generation

Abstract: Representation-alignment methods effectively accelerate the diffusion transformer training. REPA, for example, uses a pretrained vision encoder to supervise an intermediate layer through an MLP projection head which is discarded after training. It aims to import semantics from the vision encoder to the diffusion backbone. While this objective is achieved at early layers by back-propagating the alignment loss, there is no guarantee that later layers, supervised by the velocity prediction loss alone, can well access such semantics. We propose a simple solution, representation conditioning (RECON), to improve semantics in later layers. During training, we feed the pretrained vision encoder tokens into later layers; during inference, because we no longer have the clean image, we reuse the MLP head (as a proxy to the vision encoder) and project its output tokens to the later layers. Our method adds no parameters and leads to much faster convergence: RECON is $63\times$ faster than SiT and $25\times$ faster than REPA. Code is available at https://github.com/saeidrazavi/RECON

• 1. The paper proposes Representation Conditioning (RECON): a method to retrain the pipeline for the research and generate the diffusion recommendation and conditioning of the model positioning rather than discard the pre-trained projection module of transformer features, enabling more efficient and semantically aligned image generation by improving image quality through semantic conditioning.
• 2. Empirical evidence suggests that RECON speeds up convergence during training, reducing the number of necessary iterations to reach a specific quality level and increasing generation quality over siamese networks, Necessary Information for Training- Conditional Image Generation Framework (classic REPA + DINOv2) by up to 63 times while maintaining or improving upon existing evaluation metrics such as gFID.
• 3. Across various experiments, RECON consistently outperforms the standard models with a substantial improvement on both speedup, image quality metrics and semantic alignment of the image generation. The hybrid approach improves the FID (Frèchet Inception Distance) by 10%-90%'s, improving image generation with more semantically aligned representations across ImageNet and text-to-image experiments.

Problem setting and central claim

Representation alignment has become an effective strategy for accelerating diffusion-transformer training. REPA aligns an intermediate diffusion representation with features from a frozen vision encoder, such as DINOv2, using a trainable MLP projection head. The alignment loss accelerates semantic learning in the early layers, but the projection head is discarded after training. “Representation Conditioning for Generation” argues that this standard practice discards the most semantically aligned component of the model and leaves the later diffusion layers without direct access to the representation learned through alignment (2610.20608).

The paper’s central claim is therefore architectural rather than loss-based: representation alignment is more useful when the aligned representation remains accessible during generation. The proposed method, Representation Conditioning (RECON), retains the REPA projection head and uses its output as a semantic condition for the later layers of the diffusion transformer. RECON does not introduce a new training objective, and, for the B, L, and XL configurations studied, it adds no trainable parameters.

The motivation rests on three empirical observations. First, higher alignment between diffusion features and the external vision encoder correlates with lower generation FID. Second, the output of the projection head is more aligned with the external encoder than any individual diffusion-transformer layer. Third, existing alignment methods discard this projection head after training. RECON converts the projection head from a training-only mechanism into an inference-time source of semantic conditioning.

[Figure 1]

Figure 1: RECON retains the projection head and conditions later diffusion layers on semantic tokens derived from aligned representations.

Method

The experiments use SiT-style flow matching. Given a VAE latent $z_0$ and Gaussian noise $z_1$, the model receives an interpolated latent $z_t$ and predicts the velocity $z_1-z_0$. A diffusion transformer is divided conceptually at an alignment depth $\ell$ into early layers, which produce the representation aligned with the vision encoder, and later layers, which produce the velocity prediction.

REPA trains a projection head $g_\phi$ to map the intermediate representation $h^{(\ell)}$ into the feature space of a frozen encoder $f_{\mathrm{ext}}$. The alignment objective maximizes patchwise cosine similarity between $g_\phi(h^{(\ell)})$ and the clean-image features $f_{\mathrm{ext}}(x_0)$. RECON preserves this setup but changes how the later layers receive information.

During training, the clean-image vision-encoder features are pooled into a small number of semantic tokens. These tokens are mapped into the transformer hidden dimension and concatenated with the patch tokens after the alignment depth. The later blocks process both streams jointly, allowing patch tokens to attend to semantic tokens at every subsequent layer. The semantic tokens are removed before the final block.

During inference, clean images and their external features are unavailable. RECON consequently uses $z_1$0 as a proxy for the vision encoder, pools its output into semantic tokens, and injects those tokens into the later layers. This creates a train–inference discrepancy: training uses the external encoder’s features, whereas inference uses the projection head’s predictions. The paper addresses this discrepancy by injecting teacher-forced semantic tokens during training only for $z_1$1, with the default $z_1$2. At inference, however, semantic tokens are injected at every timestep because this empirically performs better.

The default configuration uses one pooled semantic token, DINOv2-B as the target encoder, alignment depth 8 for SiT-XL/2, and zero-padding for dimension matching when the target representation is smaller than the diffusion hidden dimension. Zero-padding is parameter-free. Only the Small configuration requires a learned linear mapping, with fewer than one million additional parameters.

Representation analysis

The paper evaluates semantic alignment using CKNNA against DINOv2-B and linear probing on ImageNet. The projection-head output has the highest CKNNA with the target encoder, exceeding the alignment of every diffusion-transformer layer. This result is structurally expected because the projection head is directly optimized against the target representation, but its practical implication is important: the most aligned feature is precisely the feature that REPA discards.

RECON improves semantic alignment throughout the diffusion backbone, not merely in the appended semantic tokens. When semantic tokens are excluded from evaluation, the later layers receiving those tokens exhibit a mean CKNNA improvement of 21.2% and a mean linear-probing improvement of 4.9% relative to REPA. The result supports the paper’s mechanism: conditioning does not simply attach an external semantic shortcut to the output; it changes the representations computed by the later diffusion blocks.

[Figure 2]

Figure 2: RECON increases alignment with the external vision encoder across the diffusion-transformer layers.

The paper also reports a correlation between CKNNA and generation quality across training checkpoints. Checkpoints with stronger alignment tend to have lower gFID, linking representation-level measurements to sample-level performance. This correlation does not establish that alignment is the sole causal factor governing generation quality, but it motivates using semantic alignment as a diagnostic for diffusion training.

RECON also changes the temporal structure of generation. The authors measure the semantic similarity between the running clean-image estimate $z_1$3 and the final generated image. Across 10,000 samples, RECON reaches a fixed semantic similarity threshold earlier than REPA. In heavily corrupted-image experiments, with inputs containing 80% noise and 20% image signal, RECON recovers subject identity and global composition earlier and more reliably than REPA.

[Figure 3]

Figure 3: RECON causes subject and layout information to stabilize earlier along the sampling trajectory.

This earlier semantic commitment has a direct computational consequence. At 400K training iterations, REPA’s gFID increases from 7.9 at 250 sampling evaluations to 8.4 at 100 and 10.8 at 50. RECON maintains gFID near 3.2 at both 100 and 250 evaluations and reaches 3.3 at 50. Thus, under this evaluation protocol, RECON reduces inference network evaluations by approximately 2.5–5 times with little degradation in gFID.

Class-conditional generation

On ImageNet at $z_1$4 resolution without classifier-free guidance, RECON substantially improves convergence relative to both SiT and REPA. With SD-VAE latents, SiT-XL/2 plus RECON obtains gFID 3.25 after 400K iterations, compared with 5.90 for REPA at the same iteration count. At 1M and 2.4M iterations, RECON reaches gFID 2.54 and 2.21, respectively.

The paper emphasizes the training-speed advantage. Its headline comparison reports that RECON is 63 times faster than SiT and 25 times faster than REPA in reaching comparable generation quality (2610.20608). The exact speedup depends on the selected quality threshold and comparison protocol, but the underlying convergence gap is clear: RECON reaches quality levels that conventional diffusion transformers and REPA require substantially more optimization to attain.

Method or setting	Training budget	gFID
SiT-XL/2	7M iterations	8.30
REPA	400K iterations	5.90
RECON	100K iterations	9.25
RECON	200K iterations	4.64
RECON	400K iterations	3.25
RECON	1M iterations	2.54
RECON	2.4M iterations	2.21

The improvement persists under guided system-level comparisons. At 80 epochs, RECON reaches FID 2.09 with the paper’s improved Autoguidance procedure, compared with 2.28 for MaskDiT and 2.27 for DiT, despite those baselines being trained for 1600 and 1400 epochs, respectively. At 480 epochs, RECON reaches FID 1.49, comparable to scheduled-guidance REPA at 1.42 after 800 epochs and better than standard SiT at 2.06 after 1400 epochs.

[Figure 4]

Figure 4: RECON improves visual quality at matched sampling stages and reaches coherent image structure earlier than REPA.

A notable observation is that conventional classifier-free guidance is less beneficial for RECON than for ordinary diffusion transformers. The authors attribute this to the semantic richness already supplied by the conditioning tokens: the unconditional and conditional predictions become closer, leaving less conditional information for classifier-free guidance to add. They therefore use Autoguidance, together with Projected Autoguidance and Semantic Token Extrapolation, to improve guided results. These sampling modifications reduce FID from 2.26 to 2.09 at 80 epochs and from 1.80 to 1.49 at 480 epochs.

Text-to-image generation

RECON is evaluated in two text-to-image settings: MMDiT trained on MS-COCO and NanoGen trained on the substantially larger BLIP3-o dataset. In both settings it outperforms REPA on the primary metrics.

On MS-COCO with guidance scale $z_1$5, RECON reduces FID from 9.85 to 8.26 and increases CLIP score from 0.2273 to 0.2313. With $z_1$6, FID improves from 4.73 to 4.55 and CLIP score from 0.2443 to 0.2462. On BLIP3-o, the improvement at $z_1$7 is larger: FID decreases from 5.41 to 3.90, while Inception Score increases from 22.40 to 24.31. At $z_1$8, RECON retains a smaller but positive advantage, reducing FID from 5.30 to 5.04 and increasing Inception Score from 27.61 to 27.91.

[Figure 5]

Figure 5: RECON produces stronger text-to-image samples than REPA under matched training budgets.

These results indicate that the method is not restricted to class labels or to ImageNet’s relatively compact semantic structure. Nevertheless, the text-to-image experiments compare models trained under particular architectures, datasets, and training schedules; they establish robustness across the reported settings rather than universal superiority across all text-conditioned diffusion systems.

Generalization and ablations

RECON improves REPA across model scale. After 20 epochs, the gFID of SiT-L/2 with RECON is 11.24, compared with 18.88 for SiT-XL/2 with REPA. Thus, a smaller model equipped with representation conditioning can outperform a larger REPA model at the same training duration. For SiT-B/2, SiT-L/2, and SiT-XL/2, RECON improves gFID from 46.35 to 26.08, 22.95 to 11.24, and 18.88 to 9.25, respectively.

The method also generalizes across tokenizers. At 80 epochs, RECON changes gFID from 7.84 to 3.25 with SD-VAE, from 7.41 to 4.78 with InVAE, and from 4.95 to 4.02 with VA-VAE. The smaller gain with VA-VAE is consistent with its already semantically structured latent space, which includes an additional representation-alignment objective. This result suggests that RECON is most valuable when the tokenizer does not itself provide strong semantic organization.

Across five target encoders—DINOv2-B, I-JEPA-H, MoCo-v3-B, SigLIP2-B, and WebSSL-DINO-1B—RECON consistently improves over REPA. The method therefore does not appear to depend narrowly on DINOv2, although the magnitude of improvement varies across encoders.

The ablations identify several important design choices. A single semantic token performs better than four tokens: increasing the token count from one to four raises gFID from 3.25 to 4.70. The authors attribute this degradation to the amplification of projection errors across multiple predicted semantic tokens. The default injection threshold $z_1$9 also matters. Injecting teacher-forced tokens only for $z_t$0 produces gFID 5.33, whereas injecting them for all training timesteps produces 4.70. The intermediate threshold gives 3.25, indicating that both excessive reliance on unavailable clean features and insufficient semantic supervision are harmful.

Teacher forcing is particularly important. At 100K iterations, training semantic tokens from the clean-image encoder produces gFID 9.2, whereas constructing them online from $z_t$1 produces 17.1; using an exponential moving average of the projection head only improves this to 16.8. The projection head is too inaccurate and unstable early in training to serve as a reliable teacher for the later layers.

An apparent contradiction concerns inference-time injection. Training suppresses semantic-token injection at high noise levels to reduce the train–inference gap, yet injecting tokens at every inference timestep performs better than restricting injection to the training interval. In the reported ablation, all-timestep inference gives gFID 3.2 versus 6.3 when injection is restricted to $z_t$2. This indicates that the later layers can exploit semantic tokens outside the regime in which they were explicitly trained, although the mechanism behind this extrapolation remains unresolved.

Limitations and open questions

RECON requires a projection head trained by a REPA-style representation-alignment objective. It cannot be directly applied to a model trained without such alignment. Consequently, the method eliminates the waste of discarding the projection head but does not eliminate the need for an external or otherwise learned semantic target.

The principal quality limitation is a fidelity–diversity trade-off. At 80 epochs with SD-VAE, RECON increases precision from 0.70 to 0.81 and Inception Score from 125.4 to 204.2, but decreases recall from 0.65 to 0.56. The method therefore produces more semantically decisive and class-consistent samples at the cost of reduced coverage of the data manifold. This trade-off is also observed in representation-rich latent approaches such as RAE and is not resolved by the proposed conditioning mechanism.

The use of teacher forcing leaves a nontrivial distribution gap between training and inference. The scheduling heuristic reduces this gap but does not remove it, and the fact that all-timestep inference works better than schedule-matched inference suggests that the current analysis is incomplete. The paper also leaves open why RECON’s improvement is substantially smaller for VA-VAE, whether a richer semantic-token construction would improve already semantic latent spaces, and whether the observed gains are limited by the tokenizer rather than by the diffusion backbone.

Finally, the evaluation is concentrated on ImageNet and two text-to-image configurations. The results establish consistency across several model scales, tokenizers, encoders, and conditioning modalities, but they do not determine how RECON behaves under substantially different resolutions, larger multimodal backbones, or target representations whose geometry is poorly matched to the generation task.

Conclusion

“Representation Conditioning for Generation” identifies a specific inefficiency in representation-aligned diffusion training: the projection head that most directly encodes the target semantics is discarded before sampling. RECON retains that head and exposes its output to later diffusion layers through pooled semantic tokens. The intervention is architecturally small, parameter-efficient, and compatible with existing REPA-style training.

Across ImageNet and text-to-image experiments, RECON improves convergence, semantic alignment, generation quality, and sampling-step efficiency. The strongest reported results include gFID 3.25 after 400K iterations, a 21.2% improvement in later-layer CKNNA, and near-constant gFID when reducing sampling from 250 to 50 evaluations. Its limitations are equally concrete: dependence on representation alignment, reduced recall, a teacher-forcing discrepancy, and weaker gains for already semantic tokenizers. The paper’s principal contribution is consequently a precise design principle: acquiring aligned representations is insufficient if the generator cannot access them where the final prediction is formed.