ResDiT: Training-Free HR Diffusion Transformer

• ResDiT is a training-free restructuring methodology for Diffusion Transformers that enables high-resolution image synthesis by preserving the pre-trained model’s generative capabilities.
• It employs positional embedding scaling and a patch-based local attention mechanism to maintain spatial layout integrity and recover high-frequency texture details.
• The approach fuses global and local features using Gaussian-weighted splicing and Fourier-domain techniques, achieving competitive metrics in HR generation.

ResDiT is a training-free restructuring methodology for Diffusion Transformers (DiTs) that enables high-resolution (HR) image synthesis directly from pre-trained models, circumventing layout collapse and fidelity degradation issues inherent to naïve extrapolation. The approach exploits intrinsic generative mechanisms rather than multi-stage or cascaded pipelines, activating latent spatial scalability through specific architectural augmentations. At the core, ResDiT identifies the role of positional embeddings (PEs) in spatial layout integrity and designs a path to efficient HR generation via PE scaling, local enhancement, and fusion strategies, thus preserving both global coherence and fine texture detail far beyond the native training resolution of existing DiTs (Ma et al., 1 Dec 2025).

1. Problem Formulation and Limitations of Conventional Scaling

Diffusion Transformers are typically trained to denoise latent representations $x_t \in \mathbb{R}^{h\cdot w \times c}$ at a fixed spatial resolution $h \times w$. Each denoising step uses self-attention mechanisms and positional encodings (PEs) to translate noise $x_t$ into an estimate of $x_0$ or $\epsilon$, following standard diffusion rules:

$$x_{t-1} = \sqrt{\alpha_t} x_t + \sqrt{1-\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

At inference, substituting the latent resolution with higher dimensions ($H\times W$) causes:

• Spatial layout collapse: Objects are mis-placed or incorrectly scaled.
• Texture fidelity degradation: Details rendered at high resolutions become blurred.

These issues stem from the extrapolation of PEs and expanded global attention outside the original training domain. Vanilla DiTs apply the same denoiser parameters $\theta$, embedding function $E(\cdot)$, and self-attention $A$ to $x_t^H \in \mathbb{R}^{H\cdot W \times c}$, leading to misaligned positional signals and inefficacious receptive fields.

2. Position Embedding Scaling: Theory and Mechanism

Traditional PEs (e.g., RoPE, learned 2D coordinates) are domain-bound to $[0, h)\times[0,w)$. When directly extrapolated to $[0, H)\times[0, W)$, PE values fall outside the training range, degrading spatial localization.

ResDiT’s PE-Scaling Algorithm:

1. Compute scale factors:
   • $s_h = H/h$, $s_w = W/w$
2. For token positions $$(p_h, p_w) \in \{0, ..., H-1\}\times\{0, ..., W-1\}$$:
   • Map back to base domain: $p_h' = p_h / s_h$, $p_w' = p_w / s_w$
3. Use:

$E_H(p_h, p_w) = E_\text{base}(p_h', p_w')$

1. Only coordinate values from $[0, h)\times[0, w)$ are exposed, preserving object layout integrity.

This scaling ensures that positional encodings invoked at inference match those in the model’s training exposure, thereby eradicating global structure collapse.

3. Local-Enhancement Mechanism via Patch Attention

PE-scaling restores global layout, but HR predictions remain bereft of high-frequency detail due to the enlarged receptive field from global self-attention at $H\times W$. DiTs lack strong inductive bias for fine texture modeling at scales not seen during training.

ResDiT introduces a local patch attention branch:

• Partition $H\times W$ tokens into $N$ overlapping patches, each of size $\approx h\times w$.
• Perform self-attention within each patch independently:

$$\begin{align*} Q_i &= x^{(i)} W_Q \ K_i &= x^{(i)} W_K \ V_i &= x^{(i)} W_V \ \text{Attn}_i &= \text{softmax}(Q_i K_i^\top / \sqrt{d_k})V_i \end{align*}$$

• The outputs $\{\text{Attn}_i\}$ are spliced back into the global feature map, reinstating local texture cues.

This dual-branch arrangement recovers strong local interactions salient in the base training resolution.

4. Patch-Level Fusion and Artifact Suppression

ResDiT merges global and local branches with two fusion submodules:

A. Minimum-Overlap Partitioning: Slices spatial axes into $N$ patches (size $h$), each starting at $t_k = (k-1)\cdot(H-h)/(N-1)$ for full coverage with minimal overlap and $O(N)$ complexity.

B. Gaussian-Weighted Splicing: For positions covered by multiple patches, blend patch features using Gaussian weights centered at $c_i$:

$$w_i(p) = \exp\left(-\frac{||p-c_i||^2}{2\sigma^2}\right)$$

Aggregated output:

$$fused(p) = \frac{\sum_{i\in \mathcal{W}(p)} w_i(p) f_i(p)}{\sum_{i\in \mathcal{W}(p)} w_i(p)}$$

Eliminates grid seams and artifacts at patch boundaries.

C. Patch-Wise Spectral Fusion: Fuse low-frequency (layout) and high-frequency (texture) bands in Fourier domain per patch:

$$\begin{align*} \hat{h}_g^i &= \mathcal{F}(x_g^i), \quad \hat{h}_l^i = \mathcal{F}(x_l^i) \ x_{fused}^i &= \mathcal{F}^{-1}\left[M \odot \hat{h}_g^i + (1-M) \odot \hat{h}_l^i\right] \end{align*}$$

$M$ is a binary mask separating frequency bands. Spliced by Gaussian weights to recombine a seamless $H\times W$ output.

5. Empirical Evaluation and Comparative Performance

ResDiT was validated on a 500-prompt HR generation benchmark at $3072\times 3072$ and $4096\times 4096$ using KID, IS, CLIP-score, patch-based metrics, and user studies.

Method	KID	IS	CLIP	User Score
ResDiT	0.0189	12.91	32.85	4.8/5
Demofusion	--	--	--	--
DiffuseHigh	--	--	--	--
I-Max	--	--	--	--
HiFlow	--	--	--	--

ResDiT achieves optimal values for key metrics (best KID, IS, CLIP, and user preference) at $3072\times 3072$, outperforming representative baselines. For $4096\times 4096$, metrics drop slightly but remain competitive, and user/CLIP scores are preserved. Ablations confirm that PE-scaling, patch-based local attention, spectral fusion, and overlap-weighted splicing are all critical contributors.

6. Integration with Downstream Architectures and Extensions

ResDiT’s methodology is architecture-agnostic and can be incorporated into structured pipelines such as ControlNet, supporting structure-driven generation (e.g., semantic depth or edge maps with text prompts) at HR without accuracy loss.

PE scaling and patch partitioning generalize to arbitrary aspect ratios, allowing generation at non-square geometries such as $2048\times4096$ or $4096\times2048$ with similar fidelity.

A plausible implication is that ResDiT unlocks latent scaling potential in DiTs, supporting applications in technical and creative domains where extreme spatial resolution and precise layout control are essential, without requiring retraining or cascaded denoising guidance (Ma et al., 1 Dec 2025).

7. Technical Summary and Context

ResDiT reconstructs the resolution scalability of Diffusion Transformers by:

• Scaling positional embeddings to remain within familiar coordinate regimes.
• Reintroducing base-resolution local attention through patchwork self-attention.
• Merging global and local output branches via Gaussian-weighted patching and Fourier-domain fusion.

This synthesis restores global structure and rich detail in high-resolution outputs, functioning as a fully training-free two-branch augmentation, and surpasses methods dependent on multi-phase denoising or complex cascades. All components are vital for optimal HR generation according to comprehensive ablation studies.

ResDiT thus defines a new paradigm for efficient, scalable image synthesis with pre-trained generative transformers, adaptable to diverse architectural and task-specific contexts (Ma et al., 1 Dec 2025).