From SRA to Self-Flow: Data Augmentation or Self-Supervision?

Abstract: Representation alignment has become an effective way to accelerate diffusion transformer training and improve generation quality. Recent self-alignment methods, such as SRA and Self-Flow, further remove the dependency on external pretrained encoders by constructing alignment within the diffusion model itself. However, the mechanism behind the improvement from SRA to Self-Flow, dual-time scheduling, remains under-examined: Self-Flow attributes its gain to interactions between tokens at different noise levels, where cleaner tokens help infer noisier ones. In this work, we revisit this explanation and ask whether the gain instead comes from data augmentation along the noise dimension. To disentangle these factors, we introduce Attention Separation, which preserves the same dual-timestep input as Self-Flow while blocking attention between tokens assigned to different noise levels. Surprisingly, removing such interaction does not degrade performance and can even improve it, suggesting that the improvement from SRA to Self-Flow mainly comes from data augmentation. Furthermore,We show that Attention Separation itself provides an augmentation effect by splitting a single image into multiple effective training parts to expand the training data. Based on these observations, we combine self-representation alignment with dual-timestep and attention-separation augmentation, and demonstrate the effectiveness of this design on ImageNet.

• The paper demonstrates that noise-state data augmentation, not cross-noise self-supervision, is the primary driver behind performance improvements in Diffusion Transformers.
• It introduces Attention Separation to isolate dual-timestep effects, confirming that augmenting noise diversity enhances image generation quality.
• Experimental results indicate that balancing token partitioning with full-image sampling effectively mitigates training–inference discrepancies.

Disentangling Data Augmentation and Self-Supervision in Diffusion Transformers: An Analysis of SRA, Self-Flow, and Attention Separation

Introduction

The acceleration of training and performance improvements in Diffusion Transformers (DiTs) for image generation have been strongly influenced by advances in representation alignment. While initial approaches performed external alignment via pretrained discriminative encoders, recent methodologies—specifically Self-Representation Alignment (SRA) and Self-Flow—leverage internal self-alignment to bypass reliance on external encoders. A critical contribution of this work is a systematic investigation of the mechanism responsible for the performance gains observed when transitioning from SRA to Self-Flow, particularly focusing on whether these gains are due to enhanced self-supervision through cross-noise token interactions or are fundamentally attributable to a noise-state data augmentation effect. The study introduces Attention Separation, a compositional modification that isolates these effects, and empirically demonstrates the primacy of data augmentation in the observed improvements.

Figure 1: Conceptual distinction between Self-Flow and SRA, highlighting the dual-timestep noise scheduling unique to Self-Flow.

Background and Methodological Context

Representation Alignment Paradigms

Early work, such as REPA, adopted feature alignment between DiT outputs and external vision encoders (e.g., DINOv2), but this approach showed limitations regarding scalability and encoder dependency. SRA eliminated these extrinsic requirements by aligning student and (EMA) teacher internal representations at differing noise levels and network depths. This mechanism is intended to reinforce model-internal representation learning under noise perturbation.

Self-Flow extended this idea by implementing dual-timestep scheduling—mixing tokens at two distinct noise levels within a single input. This design was originally postulated to enhance training by enabling cleaner tokens to provide contextual guidance to their noisier counterparts, engendering stronger self-supervised learning.

Figure 2: The Attention Separation mechanism partitions tokens by timestep, thus blocking cross-timestep attention while retaining heterogeneous-noise input conditions.

Attention Separation: Isolating the Source of Gains

Mechanism and Hypothesis

Attention Separation is introduced as a controlled operation that preserves the dual-timestep token-wise noise scheduling of Self-Flow but deliberately blocks interaction (via attention masks) between tokens at different noise levels. Theoretical and empirical motivation lies in its ability to disentangle two confounded variables: 1) interaction-enabled self-supervision, and 2) the increased diversity of seen noise states per iteration, i.e., data augmentation in the noise dimension.

Figure 3: Visualization of attention patterns: standard full attention versus block-diagonal structure induced by Attention Separation in dual-timestep settings.

Empirical Findings and Analysis

Data Augmentation as the Dominant Factor

Key experiments reveal that disabling cross-timestep token attention via Attention Separation does not degrade, and in some cases enhances, generative performance (as assessed by FID and IS metrics) compared to full-attention dual-timestep training. These findings directly contradict prior hypotheses that cross-noise token interaction is essential for improvement. Instead, the augmentation of the effective sample space along the noise axis is established as the definitive driver.

Figure 4: Dual-timestep training consistently outperforms single-timestep training, regardless of the presence or absence of cross-timestep attention, affirming the data augmentation hypothesis.

Attention Separation as an Independent Augmentation Pathway

Notably, even when all tokens are assigned the same timestep (thus eliminating any cross-noise interaction altogether), partitioning the image tokens into non-interacting subsets via Attention Separation still improves performance. The plausible mechanism is that this operation increases the number of effective views or partial observations of each input, akin to spatial data augmentation.

Influence of Mask Ratio and Mitigating Training–Inference Discrepancy

Experiments varying the partitioning ratio ($\alpha$) demonstrate that excessive decomposition (large $\alpha$) can induce a training–inference mismatch, as the model is optimized on fragmented attention graphs but evaluated on fully connected ones. Mixing in a modest proportion of full-image, single-timestep samples during training alleviates this issue, restoring both FID and IS performance.

Figure 5: Introducing full-image samples corrects the performance drop at extreme Attention Separation ratios, indicating the importance of matching training and inference conditions.

System-Level Results

Quantitative evaluation on ImageNet $256\times256$ and $512\times512$ benchmarks demonstrates that combining SRA, dual-timestep scheduling, and Attention Separation yields models with FID and IS metrics that are competitive with state-of-the-art external alignment approaches while preserving the scalability and generality advantages of self-representation alignment. The method is robust across scaling regimes and achieves the best or near-best IS and FID among all tested variants.

Figure 6: Qualitative ImageNet generations with classifier-free guidance, illustrating visual fidelity resulting from the proposed training schema.

Implications and Future Directions

Theoretical Implications

This work challenges the prevailing assumption that intra-sample contextual self-supervision (via inter-timestep token interactions) is essential for improved training efficiency and sample quality in Diffusion Transformer models. Instead, it establishes data augmentation—along both the noise and spatial (part-conditioned) axes—as the primary enabler, bringing conceptual clarity to the observed gains with dual-timestep scheduling and related interventions.

Practical Implications

The proposed framework facilitates architecturally-agnostic augmentation strategies that are not contingent on specific network structures or the availability of external labels/encoders, embodying a scalable approach for large-scale generative model training. Careful balancing of token group ratios and inclusion of standard full-image attention samples are important for optimal performance and stability.

Prospects for Future Work

There is scope for systematically exploring more complex, multi-way timestep or spatial augmentations, possibly with curriculum or adaptive scheduling. Integration with advanced regularization, pretext tasks, or cross-modal alignments (e.g., joint vision-language) may yield further improvements. The insights regarding data augmentation in the noise dimension could inform the development of new generative model families beyond DiTs.

Conclusion

The transition from SRA to Self-Flow in DiT training is best explained as an effective augmentation of the training distribution along both noise and spatial dimensions, rather than as an enhancement of self-supervision via cross-noise attention. The introduction and analysis of Attention Separation delivers compelling evidence for this reinterpretation. The resulting methodology offers a principled and empirically validated path for improving sample quality and training efficiency in large-scale diffusion-based generative modeling, with implications extending to future model architectures and self-supervised training paradigms.