Post-training Quantization for Hybrid Iterative Generative Models

Abstract: Iterative Generative Models (IGMs) span autoregressive and diffusion paradigms, and hybrid variants that couple them can achieve remarkable image-generation fidelity. However, their iterative inference incurs substantial computational overhead, making Post-training Quantization (PTQ) appealing for acceleration, while directly applying vanilla PTQ to hybrid IGMs can trigger model collapse. By analyzing these failures, we identify two critical challenges: Excessive Outliers (EOs) in the activations create an irreconcilable trade-off between preserving normal precision and covering EOs, resulting in severe degradation in generation quality; Amplified Anomalies (AAs) arising unpredictably from minor quantization errors, create a mismatch between calibration and inference, thus iteratively triggering model collapse. To address these challenges, we introduce HyGenQ, a PTQ framework for hybrid IGMs. HyGenQ comprises Hierarchical Cluster Decoupling (HCD) and Scaling Recalibration (SR). HCD identifies and decouples outlier channels via a multi-stage clustering process, effectively isolating EOs while maintaining normal value precision, thereby alleviating performance degradation. SR scales AAs beyond Gaussian Bound, thereby avoiding model collapse caused by aggressive truncation. Extensive experiments demonstrate that HyGenQ successfully quantizes representative hybrid IGMs to 8-bit precision (W8A8), significantly outperforming existing baselines and validating its robustness across different model families.

• The paper introduces HyGenQ, a post-training quantization framework combining hierarchical cluster decoupling for stable outlier channels with scaling recalibration for amplified diffusion anomalies.
• HyGenQ achieves W8A8 FID scores of 2.91 on MAR-L with partially full-precision boundaries and 3.14 on fully quantized MAR-L, while competing methods often collapse with FID near 250 or higher.
• The method delivers 1.44–1.99× INT8 inference speedups using only 32 calibration samples, but sub-8-bit quantization, larger-model degradation, and scaling-recalibration overhead remain open challenges.

Motivation and problem setting

Hybrid iterative generative models (IGMs), exemplified by the Masked Autoregressive (MAR) architecture, couple token-by-token autoregressive modeling of global structure with diffusion-style iterative refinement of per-token distributions. This coupling yields high-fidelity image synthesis but imposes a severe inference cost: autoregressive context updates and multi-step DDPM denoising are both dominated by matrix multiplications in linear layers. Post-training quantization (PTQ) is an attractive remedy because it requires no retraining and only a small calibration set, yet the paper shows that vanilla PTQ applied to hybrid IGMs does not merely degrade quality—it can trigger complete model collapse.

The paper's central contribution is a diagnosis of two failure modes specific to the coupled autoregressive–diffusion inference loop, followed by a PTQ framework, HyGenQ, that addresses each mode directly.

Two failure modes: Excessive Outliers and Amplified Anomalies

Excessive Outliers (EOs) are sparse, extremely large activations that arise from three structural properties of hybrid IGMs: stepwise generation under partial context forces the autoregressive Transformer to learn sharper signals; per-token DiffLoss backpropagates noise-regression gradients into the AR backbone with reused conditioning vectors across timesteps; and the composition of two iterative processes extends the error-propagation path. Empirically, channel-wise maxima in MAR far exceed those of a ViT-B at comparable layers. Because uniform quantizer resolution is set by the activation range, EOs create an irreconcilable trade-off: preserving outliers crushes normal values into few levels, while MSE-based truncation of outliers causes irreversible quality collapse.

Amplified Anomalies (AAs) arise during quantized diffusion refinement. The paper decomposes the denoising update to show that quantization error $\Delta_{\varepsilon_\theta}$ in the predicted noise is scaled by $\gamma_t$—which is substantially larger at early timesteps in MAR than in conventional diffusion models—and injected into the next denoising state via the boundary layers (the input and output linear layers of the denoising MLP). The perturbed tokens then enter subsequent autoregressive contexts, so local rounding errors propagate and accumulate across iterations. Static calibration ranges derived offline cannot cover these unpredictable overflows; aggressive clipping then destabilizes generation and produces collapse. This calibration–inference mismatch is the key reason existing PTQ methods fail when all linear layers are quantized.

Method: Hierarchical Cluster Decoupling and Scaling Recalibration

Hierarchical Cluster Decoupling (HCD) exploits the observation that outlier channels remain highly consistent across iterations within a layer. HCD proceeds in three stages: (1) channels are clustered by their maximum absolute activations using $k$-means enhanced by a genetic algorithm to avoid poor local minima under highly skewed distributions; (2) cluster-level maxima are aggregated and partitioned into a main (normal) and non-main (outlier) group; and (3) the optimal cluster count $k^*$ is selected by minimizing a weighted Sum-of-Squared-Errors Quantization Error Metric computed via pseudo-quantization, with weight $\omega$ emphasizing outlier-channel errors. Normal and outlier channels are then quantized at the same bit-width but with independent scales and zero-points, resolving the range/precision trade-off. Detected outlier sets are fixed per layer and reused without recomputation.

Scaling Recalibration (SR) targets boundary-layer activations, which follow a standard Gaussian distribution in the full-precision model (supported by Kolmogorov–Smirnov statistics with mean 0.0211). SR defines a Gaussian Bound $G = \Phi^{-1}(\eta)$; at each timestep, channels whose maximum activation exceeds $G$ are rescaled back into the bound, with inverse scaling afterward. For moderate distributional shifts, SR adopts fixed-range quantization over $[-G, G]$ rather than sample-dependent extrema. The bound is derived analytically ($\eta \approx 0.9999999853$, giving $G = 5.545$) and used globally across all experiments, avoiding per-model tuning.

Experimental results

The evaluation uses MAR-B/L/H on ImageNet 256×256 (50,000 generated samples, 64 iterations, 100-step DDPM solver), under two settings: spc keeps diffusion boundary layers in full precision (isolating the EO challenge), while pct quantizes all linear layers (exposing AA-induced accumulation). Calibration uses only 32 samples with per-(iteration, timestep) quantization parameters assigned uniformly across all methods.

Setting	Model	Best baseline FID	HyGenQ FID	HyGenQ IS
spc W8A8	MAR-L	3.69 (OCS)	2.91	259.59
pct W8A8	MAR-L	139.90 (TaQ-DiT)	3.14	255.35
pct W8A8	MAR-H	156.23 (TaQ-DiT)	6.89	210.95

The contrast is stark: UniformQuant and RepQ-ViT collapse outright (FID up to 564.27, IS 1.00 on MAR-H), and under pct, SmoothQuant, PTQ4DM, TFMQ-DM, and OCS all collapse to FID ≈ 250 with IS near 2, while HyGenQ remains within roughly 1 point of its spc performance on MAR-B/L. These results support the paper's claim that handling outliers alone is insufficient once anomalies can be amplified through every layer.

Robustness experiments extend the framework beyond hybrids: on LlamaGen (autoregressive) and LDM-4 (diffusion), HyGenQ consistently improves over PTQ4DM, occasionally exceeding full-precision metrics (e.g., LlamaGen-XL FID 6.598 vs. 6.713). Transferred outlier-suppression methods QuaRot and HadaNorm achieve competitive spc results but collapse under pct, reinforcing that anomaly propagation—not merely outliers—is the binding constraint in fully quantized hybrid IGMs.

Ablations attribute the gains cleanly: on MAR-B, adding HCD alone improves spc FID from 10.34 to 3.50; adding SR alone improves pct FID from 252.41 to 11.34; combining both yields 3.61. Efficiency measurements with cuBLASLt INT8 GEMM show speedups of 1.44–1.99× over full precision, essentially matching PTQ4DM, though SR recalibration costs some latency on MAR-H (1.79× vs. 2.19× for PTQ4DM under pct).

Sensitivity analyses indicate stability: results vary little across cluster-number ranges, values of $\gamma_t$0, GA hyperparameters, calibration sizes (32–128 samples), and random seeds; detected outlier-channel sets have IoU and retention of exactly 1.000 across seeds and calibration splits.

Limitations and open questions

The paper is candid about several constraints. First, lower-bit settings remain largely unsolved: W8A6 degrades sharply even for HyGenQ (best FID 144.12 on MAR-B vs. 2.30 full precision), and W4A8/W4A4 cause collapse for most methods, indicating high sensitivity of hybrid IGMs to reduced activation precision. Second, residual degradation persists under pct on larger models—MAR-H reaches FID 15.64 at 32 iterations versus 5.15 under spc—so the authors note that practical usability of deeply quantized MAR-H remains limited and that anomaly handling within the PTQ framework warrants further investigation. Third, SR introduces inference-time overhead that measurably weakens acceleration gains on MAR-H; reducing this cost is left open. Finally, the method depends on the assumption that boundary-layer activations are approximately Gaussian in the full-precision model and that outlier channels are stable across iterations; whether these assumptions hold for other hybrid architectures or text-conditioned generation is not examined.

Conclusion

HyGenQ provides a targeted PTQ solution for hybrid autoregressive–diffusion models by decoupling stable outlier channels through hierarchical clustering and constraining unpredictable anomaly amplification with a Gaussian-bound rescaling mechanism. It achieves high-fidelity W8A8 quantization where prior methods collapse entirely, generalizes across autoregressive, diffusion, and hybrid families, and delivers practical end-to-end speedups. Its main open problems—sub-8-bit stability, residual pct degradation on deep models, and recalibration overhead—define the remaining gap between quantized and full-precision hybrid IGM deployment.