DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation (2511.19365v1)

Abstract: Pixel diffusion aims to generate images directly in pixel space in an end-to-end fashion. This approach avoids the limitations of VAE in the two-stage latent diffusion, offering higher model capacity. Existing pixel diffusion models suffer from slow training and inference, as they usually model both high-frequency signals and low-frequency semantics within a single diffusion transformer (DiT). To pursue a more efficient pixel diffusion paradigm, we propose the frequency-DeCoupled pixel diffusion framework. With the intuition to decouple the generation of high and low frequency components, we leverage a lightweight pixel decoder to generate high-frequency details conditioned on semantic guidance from the DiT. This thus frees the DiT to specialize in modeling low-frequency semantics. In addition, we introduce a frequency-aware flow-matching loss that emphasizes visually salient frequencies while suppressing insignificant ones. Extensive experiments show that DeCo achieves superior performance among pixel diffusion models, attaining FID of 1.62 (256x256) and 2.22 (512x512) on ImageNet, closing the gap with latent diffusion methods. Furthermore, our pretrained text-to-image model achieves a leading overall score of 0.86 on GenEval in system-level comparison. Codes are publicly available at https://github.com/Zehong-Ma/DeCo.

• The paper introduces a dual-path architecture where the DiT encodes low-frequency semantics while a lightweight pixel decoder reconstructs high-frequency details at each diffusion timestep.
• It employs a frequency-aware flow-matching loss that uses adaptive JPEG-based weighting to prioritize perceptually salient frequency bands.
• The method achieves state-of-the-art FID scores on ImageNet with significant gains in training speed and inference efficiency over traditional approaches.

DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation

Motivation and Context

End-to-end pixel diffusion models bypass the limitations of latent diffusion, namely VAE-induced compression artifacts and capacity bottlenecks, by directly modeling raw pixel distributions. However, conventional pixel-diffusion frameworks struggle with simultaneous modeling of low-frequency semantic content and high-frequency signals, the latter comprising complex details and noise. This entanglement adversely impacts both learning dynamics and final synthesis quality. "DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation" (2511.19365) addresses this bottleneck by proposing a novel architecture that explicitly decouples the modeling of these frequency components during generative modeling.

Frequency-Decoupled Architecture

The DeCo framework is constructed around a two-path architecture:

• The DiT (Diffusion Transformer) operates on downsampled input, specializing in modeling low-frequency semantic content.
• A lightweight pixel decoder, devoid of self-attention, receives semantic guidance from the DiT and is solely responsible for reconstructing high-frequency signals on the original-resolution image space.

This decoupling is realized at every diffusion timestep, permitting the two modules to focus exclusively on their optimal frequency bands and avoiding capacity dilution.

Figure 1: The DeCo architecture decouples low-frequency modeling (DiT) from high-frequency reconstruction (pixel decoder), resulting in improved semantic representation and enhanced image quality.

Figure 2: The overall framework: DiT encodes downsampled images, pixel decoder reconstructs details conditioned on DiT output.

Frequency-Aware Training Objective

To further leverage the frequency separation, DeCo introduces a frequency-aware flow-matching (FM) loss. Predicted pixel velocities and ground-truth velocities are transformed to the frequency domain (via DCT on YCbCr color space). Adaptive weights for each frequency band are derived from JPEG quantization tables—the normalized reciprocals encode human-perceptual importance, prioritizing visually salient frequencies and suppressing noise. This objective encourages the model to allocate learning resources commensurate with the impact of each frequency band in synthesis quality.

Figure 3: Illustration of JPEG base and scaled quantization tables used for perceptual frequency weighting in the FM loss.

Empirical Analysis of Frequency Decoupling

Quantitative analysis of DCT energy spectra reveals explicit separation: DiT outputs in DeCo exhibit markedly reduced high-frequency energy, while the pixel decoder maintains robust energy in these bands. In contrast, baselines retain high-frequency artifacts in the DiT output, confirming entanglement and degraded synthesis.

Figure 4: DCT spectral analysis showing DeCo’s suppression of high-frequency energy in DiT output and retention in pixel velocity prediction.

Numerical Results

DeCo achieves state-of-the-art FID for end-to-end pixel diffusion:

• ImageNet $256\times256$: FID 1.62
• ImageNet $512\times512$: FID 2.22

At $256\times256$ resolution, DeCo yields a 10x gain in training speed over baseline, attaining 2.57 FID in 400k iterations. The architecture also demonstrates high inference efficiency (1.05s/image for 100 steps), outperforming prior multi-scale cascade models and single-scale transformers across speed and synthesis quality.

Figure 5: FID learning curve comparison—DeCo converges to low FID orders of magnitude faster than baseline.

For text-to-image synthesis over the GenEval benchmark, pretrained DeCo achieves an overall score of 0.86, outperforming SD3 and FLUX.1-dev despite using the same data and text encoder as BLIP3o.

Qualitative Results

DeCo produces highly detailed and semantically consistent generations across both class-conditioned and text-conditioned tasks. The decoupling ensures sharp boundaries, realistic textures, and global semantic accuracy.

Figure 6: Example class-to-image generations at $256\times256$—exhibiting detail fidelity and semantic plausibility.

Figure 7: Multilingual text-to-image generations at $512\times512$, showing interpretability and localization at high resolution.

Figure 8: Diverse qualitative results for class-to-image at $256\times256$.

Figure 9: High-resolution ($512\times512$) class-conditioned results, displaying intricate texture synthesis.

Architectural Ablations

Systematic ablations confirm:

• Decoder hidden size of 32 and depth of 3 offer optimal tradeoff between capacity and efficiency.
• Decoder patch size of 1 (full-resolution input) is essential; patchification severely degrades high-frequency generation.
• AdaLN-based interaction between DiT and decoder is significantly more effective than additive fusion.
• JPEG quality parameter in $\mathcal{L}_{\mathrm{FreqFM}}$ optimally set to 85; lower values over-suppress detail, higher ones fail to prioritize salient frequencies.

Implications and Future Research

DeCo provides a generic paradigm for frequency-decoupled modeling in generative frameworks beyond static images, suggesting applicability to video diffusion and multi-modal synthesis. It eliminates the VAE bottleneck, making pixel diffusion models competitive with, and in some regimes superior to, latent methods. The adaptive frequency weighting mechanism is directly extensible to other perceptual tasks. Further directions involve generalized frequency-domain adaptivity, architectural variants of the pixel decoder, and investigation of task-specific quantization priors for applications in specialized domain generation.

Conclusion

DeCo introduces a principled architectural and optimization decoupling of semantic and detail frequencies in pixel diffusion, achieving efficient, high-quality end-to-end image synthesis without latent-space compression. Its strong quantitative and qualitative results validate explicit frequency separation as a foundational mechanism for scalable generative modeling. The frequency-aware FM loss offers a perceptually motivated pathway for further advances in resource allocation during model training and for extension to broader generative domains.