What matters for Representation Alignment: Global Information or Spatial Structure? (2512.10794v1)

Abstract: Representation alignment (REPA) guides generative training by distilling representations from a strong, pretrained vision encoder to intermediate diffusion features. We investigate a fundamental question: what aspect of the target representation matters for generation, its \textit{global} \revision{semantic} information (e.g., measured by ImageNet-1K accuracy) or its spatial structure (i.e. pairwise cosine similarity between patch tokens)? Prevalent wisdom holds that stronger global semantic performance leads to better generation as a target representation. To study this, we first perform a large-scale empirical analysis across 27 different vision encoders and different model scales. The results are surprising; spatial structure, rather than global performance, drives the generation performance of a target representation. To further study this, we introduce two straightforward modifications, which specifically accentuate the transfer of \emph{spatial} information. We replace the standard MLP projection layer in REPA with a simple convolution layer and introduce a spatial normalization layer for the external representation. Surprisingly, our simple method (implemented in $<$4 lines of code), termed iREPA, consistently improves convergence speed of REPA, across a diverse set of vision encoders, model sizes, and training variants (such as REPA, REPA-E, Meanflow, JiT etc). %, etc. Our work motivates revisiting the fundamental working mechanism of representational alignment and how it can be leveraged for improved training of generative models. The code and project page are available at https://end2end-diffusion.github.io/irepa

• The paper finds that spatial structure of encoder features is a more reliable predictor of generative performance than global semantic accuracy.
• It employs comprehensive correlation studies over 27 vision encoders and introduces iREPA modifications, including convolutional projection and spatial normalization.
• Results show that preserving spatial organization improves diffusion model convergence and sample quality, challenging traditional encoder selection criteria.

What Matters for Representation Alignment: Global Information or Spatial Structure?

Introduction

This work presents a comprehensive empirical analysis of representation alignment (REPA) for diffusion models, investigating the critical factors that enhance generative modeling performance when aligning diffusion features with external pretrained vision encoders. Challenging the widespread assumption that global semantic performance (e.g., ImageNet classification accuracy) of the target representation predicts generative quality, the authors demonstrate that the spatial structure of encoder features—i.e., the pairwise relationships between patch tokens—serves as a substantially more reliable determinant for generation outcomes. This stands in direct contrast to the current practice of selecting target encoders based on their discriminative (classification) power, and has substantial implications for generative modeling, especially in the selection and design of vision encoders used for guidance and alignment.

Empirical Analysis: Global Information versus Spatial Structure

The prevailing paradigm assumes that higher validation accuracy and stronger global semantic representations in vision encoders translate into superior generative performance following REPA. This paper systematically demonstrates that this assumption does not hold. Through correlation studies across 27 diverse vision encoders and multiple model scales, the authors observe that there exists a weak association between linear probe (classification) accuracy and generative metrics such as FID, and, occasionally, an inverse relationship—higher discriminative accuracy can correlate with worse generative quality.

The analysis is substantiated by multiple cases where smaller or spatially-tuned encoders significantly outperform larger, higher-accuracy counterparts in image generation (Figure 1; Figure 2), including cases with encoders like SAM2 that, despite minimal global information, yield superior FID performance when used for REPA.

Figure 1: Correlation analysis across 27 diverse vision encoders: spatial structure, not global performance, drives generation quality. Simple spatial feature transfer modifications lead to faster REPA convergence.

Figure 2: Example cases where higher classification accuracy leads to worse FID in generation. Spatial structure provides a better predictor of generation quality.

A critical supporting experiment involves modifying patch token representations by mixing in additional global components (CLS token); while this increases linear probing accuracy, it consistently degrades generation quality, with FID worsening significantly as more global information is injected.

Quantifying Spatial Structure and Its Predictive Power

To formalize the notion of spatial structure, the authors define several spatial self-similarity metrics (including LDS, CDS, SRSS, and RMSC; see detailed definitions in the text and supplementary). Each metric quantifies how similarity patterns between feature patches decay with spatial distance or semantic grouping, capturing the degree to which an encoder preserves local and region-specific relationships.

The empirical results indicate that all spatial structure metrics exhibit a high absolute Pearson correlation with generation performance (|r| > 0.85), across all encoders and model sizes, while linear probing accuracy correlates weakly (|r| ≈ 0.26) or even positively with worse FID scores.

Figure 3: Across 27 vision encoders, spatial structure metrics (LDS, SRSS, CDS, RMSC) show much higher correlation with generation quality (FID) than linear probing accuracy.

This effect holds robustly even after removing outlier architectures and across both intermediate and output-layer representations. The findings disprove the assumption that global semantic features are beneficial for generation, and instead confirm that spatial organization within encoder features enables better transfer and training acceleration in diffusion architectures.

iREPA: Architectural Modifications for Spatial Structure Preservation

Grounded on these empirical insights, the authors introduce two architectural modifications to enhance REPA, termed iREPA:

1. Conv Projection Layer: Replaces the standard 3-layer MLP projection with a lightweight convolutional layer for feature dimension mapping, leveraging the inherent spatial inductive bias of convolutions to better preserve local structure and reduce spatial information loss (Figure 4).
2. Spatial Normalization Layer: Implements a normalization operation across patch tokens, removing the mean feature (global component) and normalizing by variance. This explicitly suppresses global information, accentuates spatial contrast between tokens, and enhances local semantic coherence.

Figure 4: A convolutional projection layer better preserves spatial arrangement in transferred features than standard MLP, minimizing loss of local detail.

Figure 5: Spatial normalization transforms patch token similarity maps, removing global overlays and enhancing local spatial boundaries.

The combination of these modifications is trivial to implement—requiring less than four lines of code per the paper's reference implementation—and yields improvements across all measured configurations.

Experimental Evaluation and Implications

iREPA is systematically validated using multiple vision encoders (DINOv2, DINOv3, CLIP-L, WebSSL, Perception Encoder, etc.), model scales (SiT-B, SiT-L, SiT-XL), and various generative frameworks (REPA, REPA-E, MeanFlow, JiT). The results are unequivocal:

• iREPA consistently accelerates convergence, lowering FID and increasing IS across all encoders and scales.
• Gains scale with model size; percentage improvement is greater with larger encoders and backbones.
• Performance improvements extend to diverse training recipes and even to pixel-space diffusion models (Figure 6).

  Figure 7: Emphasizing spatial structure leads to consistent generation quality and convergence improvements, as shown for SiT-XL/2 with REPA and iREPA.

These empirical observations reverse the conventional wisdom regarding the selection of vision encoder targets for representation alignment. The primary implication is a fundamental shift in designing and pretraining vision encoders for generative tasks: priority should be placed on architectural, pretext, and loss choices that maximize spatial self-similarity, not global, image-level semantic discriminativeness.

Beyond the immediate implications for fast and stable diffusion model training, the findings suggest that leveraging classical, handcrafted spatial features (e.g., SIFT, HOG) or spatially-tuned intermediate network features, can provide strong alignment signals even if they are poor classifiers—an observation supported by experimental evidence in the main text.

Theoretical and Practical Ramifications

Theoretically, the results motivate a reconsideration of the objectives of self-supervised and supervised vision pretraining. Inductive biases and optimization strategies that enhance spatial self-similarity—possibly at the cost of classification accuracy—can offer strong inductive priors for downstream generation. The demonstrated efficacy of spatial normalization and convolution-based projection provides a prescriptive guide for future architecture design in generative modeling setups.

Practically, practitioners should favor spatially-structured vision feature targets, employ explicit spatial regularization via normalization, and avoid injecting excessive global information (e.g., through feature pooling or specialty tokens) when the intent is diffusion feature alignment and sample quality. The lightweight nature of the iREPA modifications makes them widely and immediately applicable without computational penalty.

Future Directions

These findings indicate potential new research in:

• Large-scale spatial structure–oriented encoder pretraining regimes for generative tasks, distinct from those optimized for classification.
• Metric-driven automated target encoder selection based on SSMs rather than classification proxies.
• Extending spatial structure–preserving alignment to other domains (video, 3D, multimodal) and broader architectural families.

Conclusion

The paper delivers compelling empirical and algorithmic evidence that spatial structure, not global semantic information, governs the effectiveness of representation alignment in diffusion model training. The proposed iREPA variant codifies this by making straightforward modifications to preserve and accentuate spatial arrangements during feature transfer—resulting in uniform and significant improvements across model types, training recipes, and generations. The work establishes spatial self-similarity as the primary axis of encoder selection and architecture design for generative applications, fundamentally revising the conventional criteria used for alignment targets.