CoD-Lite: Real-Time Diffusion-Based Generative Image Compression

Abstract: Recent advanced diffusion methods typically derive strong generative priors by scaling diffusion transformers. However, scaling fails to generalize when adapted for real-time compression scenarios that demand lightweight models. In this paper, we explore the design of real-time and lightweight diffusion codecs by addressing two pivotal questions. First, does diffusion pre-training benefit lightweight diffusion codecs? Through systematic analysis, we find that generation-oriented pre-training is less effective at small model scales whereas compression-oriented pre-training yields consistently better performance. Second, are transformers essential? We find that while global attention is crucial for standard generation, lightweight convolutions suffice for compression-oriented diffusion when paired with distillation. Guided by these findings, we establish a one-step lightweight convolution diffusion codec that achieves real-time $60$~FPS encoding and $42$~FPS decoding at 1080p. Further enhanced by distillation and adversarial learning, the proposed codec reduces bitrate by 85\% at a comparable FID to MS-ILLM, bridging the gap between generative compression and practical real-time deployment. Codes are released at https://github.com/microsoft/GenCodec/CoD_Lite

• The paper introduces CoD-Lite, a lightweight diffusion-based model that achieves real-time decoding at 1080p with competitive perceptual quality.
• The paper demonstrates that compression-oriented pre-training and localized attention (via convolutions) can replace expensive global transformers with minimal performance loss.
• The paper shows that integrating distillation and adversarial tuning enables up to 85% bitrate reduction compared to heavy transformer-based codecs.

Real-Time, Lightweight Diffusion-Based Image Compression: An Analysis of CoD-Lite

Motivation and Problem Space

Recent advances in neural image compression, particularly those utilizing powerful generative models such as GANs and diffusion models, have achieved high perceptual fidelity at low bitrates. However, state-of-the-art generative codecs have increasingly relied on large-scale transformer-based models, resulting in substantial computational burdens and prohibitive inference latencies, especially unsuited for real-time applications. Diffusion models, while surpassing GANs in generation quality, further exacerbate this trend, commonly employing architectures with hundreds of millions to billions of parameters that translate into decoding speeds below the real-time threshold (e.g., <3 FPS at 1080p).

CoD-Lite addresses two fundamental bottlenecks in this landscape:

1. The limited utility of standard diffusion pre-training for lightweight models under stringent resource constraints.
2. The empirical necessity of transformer-based global attention in achieving competitive generative compression, particularly within the context of image-native conditioned reconstruction.

Methodological Contributions

1. Re-assessing Diffusion Pre-training Paradigms

A central empirical investigation of CoD-Lite is whether generative (i.e., unconditional or class/text-conditioned) diffusion pre-training, effective at large scales, translates to the lightweight regime (tens of millions of parameters). Systematic experiments reveal that:

• Generation-oriented pre-training, which provides substantial FID improvements for 700M-parameter models, offers negligible benefit for 34M-parameter models, due to an information-capacity mismatch.
• Compression-oriented diffusion pre-training (as in CoD), leveraging image-native, information-rich conditions, consistently yields substantial gains across model scales, even with compact backbones. The entropy of conditioning information becomes the dominant factor when capacity is limited: more bits in the conditioning vector directly translate to better perceptual outcomes at small scale.

2. Revisiting the Role of Transformers in Diffusion-based Compression

While diffusion transformers (DiTs) are default backbones for state-of-the-art generative models, their $O(N^2)$ global attention imposes substantial latency overhead. CoD-Lite provides a detailed analysis of attention patterns in compression settings, visualizing attention in all layers of a DiT-based CoD model. This analysis demonstrates that, unlike in generation-oriented tasks, the majority of attention mass in compression models is localized, with only early, alignment-focused layers exhibiting global interactions necessary mainly for feature alignment (e.g., with DINOv2 features).

Through ablation, CoD-Lite demonstrates:

• Local window attention and even depth-wise convolutions offer near-equivalent rate-distortion performance compared to full global attention at a fraction of the inference cost.
• With appropriate distribution-matching distillation (DMD) from a large DiT-based teacher and adversarial tuning, depth-wise convolutional backbones maintain competitive FID to transformer-based counterparts.

3. System Architecture and Training Regime

The proposed CoD-Lite codec comprises:

• An encoder/decoder pipeline based on residual blocks and vector quantization, covering a wide operative bitrate range (0.0039–0.5 bpp).
• A lightweight (52M) convolutional diffusion backbone with depth-wise convolutions and channel attention, omitting unnecessary architectural elements such as AdaLN-Zero for further efficiency.
• A two-stage training process involving initial compression-oriented diffusion pre-training (via unified flow-matching loss) and subsequent distillation-guided one-step tuning with DMD loss and adversarial objective using a projected GAN.

Experimental Results

The CoD-Lite codec is benchmarked against a comprehensive suite of state-of-the-art codecs, ranging from GAN-based (MS-ILLM, TACO) to both multi-step and one-step diffusion methods (PerCo, DiffC, OSCAR, StableCodec), across standard datasets (Kodak, CLIC2020, Div2K) and evaluated with perceptual (FID, LPIPS, DISTS) and distortion (PSNR) metrics.

Key empirical findings:

• CoD-Lite achieves real-time decoding at 1080p (42 FPS) on an NVIDIA A100, a marked improvement over prior diffusion-based codecs (e.g., StableCodec at <3 FPS).
• The codec realizes an 85% bitrate reduction relative to MS-ILLM at comparable FID (Bjøntegaard Delta - BD-rate on FID).
• Visual quality at ultra-low bitrates is highly competitive, matching prior heavyweight models at a fraction of the inference complexity.
• Ablation studies rigorously demonstrate each design and training choice’s additive contribution, showing that distillation and adversarial learning are essential to bridging the quality gap induced by omitting transformers.

Implications and Theoretical Insights

Practical Impact

CoD-Lite makes real-time, high-fidelity generative image compression feasible on commercially available accelerator hardware. By decoupling high-quality perceptual compression from the requirement for massive transformer backbones, it significantly reduces deployment barriers in latency-sensitive, resource-constrained scenarios such as live media streaming or edge-based vision systems.

Theoretical Insights

The work challenges the prevailing orthodoxy around the centrality of model scale and global attention for successful diffusion-based generative compression. The findings indicate a regime-dependent optimality: as model capacity decreases, the entropy of the conditioning vector assumes primacy, and local inductive biases in convolutional backbones become not just sufficient, but preferable. The demonstrated effectiveness of distillation and adversarial guidance in this compressed regime further elucidates the role of knowledge transfer from high-capacity teachers to efficient students in generative modeling under resource constraints.

Limitations and Future Directions

While the proposed approach achieves state-of-the-art trade-offs on moderate resolutions (up to 1080p), performance at 4K and above remains limited by the training regime. Additional scaling and curriculum learning for higher resolutions, as well as exploring hybrid attentional/convolutional architectures, constitute logical next steps. Moreover, the risk of hallucinated details may restrict adoption in fidelity-critical applications (e.g., medical or forensic imaging), warranting further study.

Conclusion

CoD-Lite provides a principled, empirically validated blueprint for achieving real-time, high-fidelity diffusion-based generative image compression in resource-constrained environments. By demonstrating that compression-oriented pre-training and local convolutional backbones, augmented via distillation from large transformer-based teachers, are sufficient for both perceptual quality and inference speed, it redefines state-of-the-art capability in generative image compression. The architectural and training insights established herein are likely to inform future developments in practical, low-latency generative modeling far beyond the compression domain.