- The paper demonstrates that combining INT8 W8A8 quantization with SmoothQuant and selective bf16 protection of 17 fragile layers reliably maintains FP8-level image and text quality.
- It provides quantitative evidence that GGUF Q4_K achieves superior Pareto tradeoffs with improved Pick/CLIP scores compared to NF4 at reduced model size.
- The work emphasizes that hardware-aware post-training quantization and direct OCR evaluations are critical for deploying diffusion transformers on consumer-grade GPUs.
INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs
Introduction
This paper presents a comprehensive post-training quantization (PTQ) analysis and methodology for the 9.3B dual-branch Ideogram 4.0 text-to-image flow-matching diffusion transformer, focusing specifically on the constraints and opportunities afforded by consumer-grade Ampere RTX 3090 GPUs. These GPUs lack FP8 tensor cores but support fast INT8, motivating a systematic evaluation of INT8 and GGUF (GPTQ-based universal format) quantization strategies operating at 8-bit and sub-8-bit precision. Emphasis is placed on maintaining the FP8 quality ceiling while pushing the practical efficiency frontier, with an explicit focus on standalone and reference-aligned quality, text fidelity under quantization, and hardware-realistic memory and speed tradeoffs.
Methodology
Quantization Protocol
The quantization domain encompasses both weights and activations (W8A8) for 211 linear layers per branch in the dual-branch DiT. The Qwen3-VL text encoder and VAE remain at original precision, isolating the quantization intervention to the DiT core. FP8 is chosen as the reference due to the absence of public BF16 checkpoints.
For INT8, per-channel weight quantization is combined with per-token dynamic scaling for activations. The SmoothQuant approach (α=0.5) migrates activation outliers into the weights, mitigating accuracy loss caused by outlier-heavy activations. A rigorously-implemented fragility profiling protocol quantifies layerwise stability across denoising trajectories, pinpointing the high-kurtosis, high-variance FFN down-projections (feed_forward.w2) as critically fragile. A minimal set of 17 layers (≈8% of all linears) is protected at bf16 to restore model performance.
On the weight-only front, the paper implements and provides GGUF Q8_0 (8.5 bits/weight, naive) and Q4_K (4.5 bits/weight, group-wise) quantizers. The latter is realized via a custom NumPy/Torch path, bit-matched with the reference.
Evaluation
Benchmarking is performed on three partitioned prompt sets: calibration (n=128), quality (n=200), and text rendering/OCR (n=100, with 63 prompts supporting OCR targets). Standalone generation quality is measured by PickScore and CLIPScore. Text quality is assessed by OCR exact-match and normalized edit distance (NED) via EasyOCR. Reference fidelity is calculated against FP8 images using PSNR, SSIM, and LPIPS. All evaluations adhere to fixed seed/repro recipe for replicability.
Results
Main Findings
- INT8 W8A8 quantization with SmoothQuant and selective bf16 protection maintains the FP8 quality ceiling across both image quality and text fidelity. Paired CIs for Pick and CLIP between INT8 and FP8 include zero, confirming statistical quality parity. INT8 robustly outperforms the published NF4 (QLoRA-style) baseline by +1.9 CLIP (CI [+1.21, +2.64]), with all intervals excluding zero.
- Reference-based fidelity (PSNR/SSIM/LPIPS) places INT8 and NF4 in the same band with nuanced metric tradeoffs. INT8 yields PSNR=21.42, SSIM=0.722, LPIPS=0.306, sitting marginally below the LPIPS ≤0.30 "16-bit match" line but still within negligible difference of FP8; notably, INT8 deviates from FP8 in pixels to the same extent as NF4, but preserves strong standalone quality while NF4 incurs a real drop.
- Ablation confirms that protecting the high-fragility FFN down-projections (just 17 layers) is the dominant quality lever. Protection confers 78–92% of the CLIP recovery relative to naive W8A8 and interacts sub-additively with SmoothQuant scaling. The method’s quality plateau displays a sharp "knee" at N=17 protected layers.
- GGUF Q4_K achieves a strong Pareto optimum, outperforming NF4 in quality at identical storage: Pick/CLIP improvements of +0.96/+3.57 (CIs well clear of zero) at 10.44GB model size. Q8_0 is strictly quality-neutral relative to FP8.
Text Rendering under Quantization
OCR-based analyses (NED and exact-match) indicate that text legibility is strictly preserved by INT8 and Q4_K. INT8 records NED=0.704 vs. FP8’s 0.715 and NF4’s 0.760, with exact-match rates near zero for all (reflecting the base model's limitations on these prompts). This fine-grained OCR eval—rare in the quantized diffusion literature—demonstrates that pertinent text fidelity is decoupled from aggregate image metrics and must be directly measured.
Hardware, Efficiency, and Limitations
INT8 provides no memory or runtime advantage over FP8 absent a fused INT8 GEMM kernel (both occupy 18.6GB; INT8 incurs ≈6% higher latency than FP8 and ≈12% more than NF4). Q4_K, running at 203s/image, delivers immediate memory savings. Practical realization of INT8 speedup on Ampere-level GPUs depends on custom native kernels (e.g., CUTLASS or ViDiT-Q-style), which is explicitly identified as next-step work.
Implications
The results provide a rigorous roadmap for deploying high-performance, quantized diffusion transformers on consumer hardware, underlining the importance of hardware-aware, PTQ methodology over theoretical quantization “aggressiveness.” The finding that near-minimal protection of high-fragility layers suffices to recover FP8-level performance is actionable for further compression and deployment studies. Furthermore, the OCR-based evaluation protocol emphasizes that standalone and perceptual distance scores alone are unreliable for text-to-image deployment scenarios—direct functional metrics are required for deployment-critical use cases such as signage generation, document synthesis, and text-centric prompt following.
On the theoretical side, the fragility analysis reiterates the necessity of module granularity in quantization interventions—a small protected set can close the gap otherwise attributed to the inherent limits of 8-bit precision. This reinforces the modular view established by prior works on quantized attention/output projections and highlights the extensibility of “quality protection” to other generative transformer architectures.
Future Prospects
The study positions fused INT8 compute kernels as the next crucial step for practical quantization gains on consumer GPUs that lack FP8 hardware. Other directions include systematizing step- and trajectory-based denoiser caching for flow-matching DiTs (e.g., via TACache strategies) and extending the reference-fidelity evaluation battery to additional quantization levels and components (text encoder, VAE) while emphasizing per-category OCR benchmarking. Broader use of alternative standalone quality judges may further disentangle pixel-based and perceptual text quality effects.
Conclusion
This work establishes that precise, selectively-protected post-training INT8 quantization allows Ideogram 4.0’s 9.3B diffusion transformer to match the FP8 quality ceiling, including on quantization-sensitive text rendering, and robustly outperforms NF4 at fixed configuration. On the quality–memory Pareto frontier, GGUF Q4_K surpasses NF4 in both axes. Realizing INT8’s practical speed advantage on consumer GPUs will require targeted kernel engineering, for which the present protocol provides a high-performing substrate. The reproducible releases and protocol offer the community a strong platform for deployment and further quantized model research.
Reference:
"Holding the FP8 Quality Ceiling at 8-Bit Weights and Activations: INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs" (2606.12280)