SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers

Abstract: Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. However, these strategies apply a uniform and content-agnostic scaling across RoPE components with distinct frequency characteristics, inducing a trade-off between preserving global structure and recovering fine detail. We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent's spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity. Experiments show that SEGA consistently improves high-resolution synthesis across multiple target resolutions, outperforming state-of-the-art training-free baselines.

• The paper introduces a novel spectral-energy guided attention scaling mechanism that resolves trade-offs between global structure and local detail in diffusion transformers.
• It employs dynamic per-dimension scaling based on latent spectral decomposition, leading to significant improvements in metrics like FID, CLIP Score, and MUSIQ.
• SEGA enables robust high-resolution synthesis in text-to-image models without retraining, demonstrating generalizability across various backbone architectures.

SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers

Introduction

Diffusion Transformers (DiTs) have become fundamental in advancing text-to-image (T2I) generative models due to their superior performance at standard training resolutions. However, these models suffer substantial performance degradation when extrapolated to resolutions beyond their training range, notably losing structural coherence and fine detail. Existing inference-time extrapolation methods generally manipulate Rotary Position Embeddings (RoPE) and attention scaling, but uniformly scale RoPE components regardless of their spatial frequency, resulting in an inherent trade-off between global structure and local detail. The paper "SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers" (2605.22668) introduces SEGA, a training-free, content-aware approach which dynamically scales attention based on the latent's spectral-frequency structure at each denoising step, thereby resolving this trade-off and enabling robust, high-fidelity synthesis across a wide range of resolutions.

Figure 1: SEGA enables high-resolution image generation with pretrained T2I models, such as Flux and Qwen, preserving both global and local structure even at extreme resolutions.

Background: High-Resolution Extrapolation and RoPE

High-resolution synthesis in DiTs is constrained by the token grid's expansion at inference, which drastically alters the distribution of positional encodings compared to the training regime. RoPE encodes spatial positions via angular rotations in embedding space at distinct frequencies. When extrapolated naïvely, the mismatch in positional offsets causes the attention to diffuse excessively, leading to blurred textures and failure of semantic consistency.

Direct extrapolation approaches, e.g., Position Interpolation, NTK-aware scaling, and YaRN, primarily adjust RoPE frequency schedules or apply uniform scaling to all RoPE components. While such modifications improve attention sharpness, they remain agnostic to the evolving spatial-frequency content of the latent, thus failing to optimally allocate focus across structural and textural elements.

Figure 2: Trade-offs in static attention scaling at $4096^2$ — low-frequency RoPE components support structure, high-frequency ones enable detail. Uniform scaling induces artifact trade-offs.

The SEGA Method

SEGA addresses the limitations of content-agnostic extrapolation by introducing adaptive, frequency-aware attention scaling grounded in the spectral-energy distribution of the latent representation at each denoising step:

• Spectral Decomposition: At each step, the latent is averaged across channels and mapped to the frequency domain via 2D FFT. Axis-aligned and radial spectral profiles are determined, representing energy distribution across frequencies for both image axes and isotropically.
• Per-Dimension Scaling: Each RoPE dimension, corresponding to a particular spatial wavelength, receives an individualized scaling magnitude ($m_d$) based on the energy localized at its frequency band. Bands with below-average energy are upscaled to augment structural cues, whereas bands with surplus energy are downscaled to prevent over-sharpening.
• Global Amplitude Modulation: The spectral flatness (Wiener entropy) of the latent spectrum quantifies the concentration of energy; this dynamically controls the global scaling amplitude. Early in denoising, when the spectrum is flat (noise-like), scaling is suppressed; as structure emerges, scaling is activated.
• Adaptive Formulation: The overall scaling for each RoPE dimension at each step is $m_d = m_{ref} \cdot \mathcal{M}_d(\mathbf{Z})$, where $m_{ref}$ grows with extrapolation ratio, and $\mathcal{M}_d(\mathbf{Z})$ is the normalized, entropy-gated content-aware correction.

  Figure 3: SEGA scaling maps highlight individualized attention scaling patterns for each RoPE component during denoising, varying by prompt and image content.

Analysis and Mechanistic Insights

SEGA's dynamism is evidenced by its impact on attention focus evolution:

• Content-Adaptivity: As shown by the evolving scaling maps and 2D spectral profiles, SEGA allocates attention precision according to where structural and textural energy naturally accumulates in the latent, rather than by predetermined heuristics.
• Enhanced Attention Structure: Comparative visualizations of attention maps for central and peripheral tokens indicate that SEGA prevents attention dilution and loss of spatial discrimination observed in fixed-scaling baselines such as YaRN.

  Figure 4: Attention maps across denoising steps show SEGA maintains sharper, more localized attention compared to uniform scaling alternatives.

Empirical Evaluation

SEGA was evaluated on both Flux and Qwen diffusion transformer models, spanning a broad regime of extrapolated resolutions ($2048 \times 4096$ up to $6144^2$):

• Quantitative Results: Across all standard and zero-shot benchmarks, SEGA surpasses baselines (NTK, YaRN, DyPE, UltraImage, guidance-based methods) on FID, CLIP Score, ImageReward, and MUSIQ. At $4096^2$, SEGA consistently demonstrates stronger semantic alignment, visual quality, and robustness, maintaining high performance as extrapolation factors increase.
• Qualitative Results: High-resolution generations, including extreme $5120^2$ and $6144^2$ settings, reveal that SEGA preserves fine details, avoids repetitive artifacts, and maintains large-scale composition, outperforming both single-stage and multi-stage guidance competitors.

  Figure 5: Across challenging prompts and multiple architectures, SEGA yields higher structural coherence and detail fidelity at $m_d$0 than state-of-the-art baselines.

  

  Figure 6: SEGA robustly maintains content geometry across non-square aspect ratios ($m_d$1, $m_d$2), mitigating distortions common to prior methods.

  

  Figure 7: In zero-shot benchmarks, SEGA generalizes to prompts exhibiting complex geometry, reflections, and lighting, handling challenging content without prompt-specific tuning.

Ablations and Design Choices

Comprehensive ablation validates the necessity of SEGA's spectral-energy guidance and design:

• Fixed vs Dynamic Scaling: Static, fixed RoPE scaling, even when tuned, underperforms SEGA, especially as resolution increases. Restricting SEGA to either axis- or global-only scaling degrades quality, confirming the complementarity of both mechanisms.
• Baseline Scaling Law: Empirical comparison of power-law and logarithmic growth for $m_d$3 indicates the need for more aggressive scaling at ultra-high resolutions, favoring power-law anchoring for stability and consistency.

Implications and Future Research

SEGA demonstrates that frequency- and content-aware adaptation at inference unlocks substantial capacity in frozen T2I diffusion transformers, making resolution extrapolation viable without retraining, multi-stage complexity, or architectural changes. Its success suggests several broader implications:

• Generalizability: SEGA's approach is agnostic to the underlying backbone (Flux, Qwen, SDXL), highlighting that latent spectral structure is a ubiquitous signal exploitable for various generative tasks.
• Resource Efficiency: By eschewing retraining, SEGA allows practical high-resolution synthesis even when massive high-res datasets are unavailable or compute is limited.
• Extensibility: The framework can be adapted for use in video, multi-modal, or sequence-based transformers where positional and frequency content are tightly coupled, addressing extrapolation challenges beyond 2D images.
• Limitations: Despite its effectiveness, SEGA depends on the quality of the underlying extrapolation (e.g., NTK), and computational requirements for multi-megapixel synthesis remain significant. At extreme scales (e.g., $m_d$4), intrinsic model capacity becomes the bottleneck.

  Figure 8: At $m_d$5, SEGA preserves high-frequency texture and structural boundaries, avoiding collapse and repetition artifacts that challenge existing RoPE-based extrapolation.

Conclusion

SEGA delivers a conceptually simple yet highly effective training-free method for stable high-resolution synthesis in DiTs by coupling RoPE attention scaling to the latent's evolving spectral-frequency structure. This spectral-energy-guided mechanism outperforms both direct-inference and complex guidance-based methods, preserving both global structure and fine detail, and generalizing across architectures and content regimes. Beyond setting a new benchmark for inference-time resolution extrapolation, SEGA motivates further research into content-aware attention adaptation grounded in frequency-domain analysis, with promising applications in modality-agnostic, scalable generative modeling.