Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation

Abstract: We present a framework for end-to-end joint quantization of Vision Transformers trained on ImageNet for the purpose of image classification. Unlike prior post-training or block-wise reconstruction methods, we jointly optimize over the entire set of all layers and inter-block dependencies without any labeled data, scaling effectively with the number of samples and completing in just one hour on a single GPU for ViT-small. We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet and, to the best of our knowledge, the first PTQ results that maintain strong accuracy on ViT, DeiT, and Swin-T models under extremely low-bit settings (W1.58A8), demonstrating the potential for efficient edge deployment. Furthermore, we introduce a data-free calibration strategy that synthesizes diverse, label-free samples using Stable Diffusion Turbo guided by learned multi-mode prompts. By encouraging diversity in both the learned prompt embeddings and the generated image features, our data-free approach achieves performance on par with real-data ImageNet calibration and surpasses simple text-prompt baselines such as "a <adjective> photo of <adjective> <cls>".

• The paper introduces an end-to-end PTQ framework that jointly optimizes quantization parameters, channel rescaling, and weight refinement across all transformer blocks using feature alignment, logit distillation, and regularization.
• The method achieves 68.45% top-1 accuracy on ViT-S at W1.58A8 with real calibration data and 63.71% using synthetic data, while prior methods collapse near random accuracy at this precision.
• Learned multi-mode prompts with Stable Diffusion Turbo generate diverse, class-consistent calibration images, typically losing only 1–2% relative accuracy compared with real data and improving over raw text prompts by up to 3.6%.

Overview

This paper presents an end-to-end post-training quantization (PTQ) framework for Vision Transformers that jointly optimizes all quantization parameters across the entire network, together with a data-free calibration strategy based on Stable Diffusion Turbo guided by learned multi-mode prompts (2602.18861). The work addresses two persistent limitations of prior ViT PTQ methods: block-wise reconstruction objectives that ignore inter-block dependencies, and dependence on real labeled calibration data. The framework achieves state-of-the-art results at W4A4 and W3A3, and, according to the authors, provides the first PTQ results at ternary weight precision (W1.58A8) on ViT, DeiT, and Swin-T backbones — a regime where prior PTQ methods such as FIMA-Q collapse to near-random accuracy.

Motivation and relation to prior work

PTQ methods developed for CNNs, including AdaRound and BRECQ, rely on layer-wise or block-wise reconstruction under statistical assumptions (approximately Gaussian activations) that do not hold for transformers. ViT activations are heavy-tailed with large outliers after softmax operations and exhibit strong inter-channel magnitude variation, breaking uniform-scale assumptions. Existing remedies each carry drawbacks: FQ-ViT uses non-uniform power-of-two quantizers with limited hardware compatibility; RepQ-ViT relies on per-layer grid search over small calibration sets and a non-standard $\log\sqrt{2}$ quantizer; APHQ-ViT replaces GELU with ReLU; FIMA-Q improves block-wise reconstruction via Fisher Information Matrix approximation but retains block-level independence. On the data-free side, prior generative approaches use hand-crafted text prompts ("a <adjective> photo of <class>"), which produce low-diversity images and fail on semantically ambiguous class names such as "kite" or "crane". The paper positions itself against all of these by formulating a single global optimization objective and by learning prompt embeddings rather than using templates.

Method

Joint quantization objective

The framework uses a standard uniform quantizer with learnable step size $\Delta$ and zero-point $z$, differentiable through the straight-through estimator. Parameters are initialized from percentile-based ranges ($P_{0.1}$ to $P_{99.9}$) computed from only 32 calibration samples, providing robustness against activation outliers. To handle channel-wise magnitude variance in transformer activations, the method introduces learnable per-channel scale ($\boldsymbol{\alpha}$) and shift ($\boldsymbol{\beta}$) vectors applied before each post-LayerNorm fully connected layer, with functionally equivalent compensation applied to weights and biases in the manner of SmoothQuant/RepQ-ViT. Unlike those methods, however, these rescaling parameters remain trainable throughout optimization rather than being fixed by grid search.

A key design choice is a zero-initialized weight refinement term added to frozen full-precision weights before quantization, allowing slight adaptation of quantized weights while remaining anchored to the original model. All parameters — step sizes, zero-points, rescaling factors, and refinement terms — are optimized jointly under a composite loss combining intermediate feature MSE alignment across all blocks, temperature-scaled KL divergence distillation of final logits ($\tau=3$), and $\ell_1$ regularization on the refinement term. No labels are used. Training runs 24,000 iterations at batch size 32 with Adam, converging in roughly one hour for ViT-Small and up to 2.5 hours for Swin-Base on a single RTX 6000 Ada GPU.

The global formulation has a practical consequence demonstrated empirically: unlike block-reconstruction methods whose accuracy saturates quickly, this approach continues to improve with larger calibration sets up to approximately 10,000 samples, particularly in ultra-low-bit settings. FIMA-Q, by contrast, cannot even be evaluated beyond 10,000 samples due to the GPU memory required to store intermediate feature maps for block reconstruction.

Data-free multi-prompt learning

For fully data-free calibration, the authors learn $M$ distinct prompt embeddings per ImageNet class (20 tokens of dimension 1024 each, matching the CLIP text encoder of SD-Turbo). Half of each prompt is initialized from the class's textual description; the remainder is randomly initialized. Prompts are optimized so that generated images are classified as the target class by a frozen pretrained ViT, with diversity enforced through three complementary regularizers: orthogonality among EOS text embeddings, and variance maximization over generated RGB images, ViT features, and attention maps. All prompts share the same latent noise during training, ensuring diversity originates solely from the learned prompts. A reinitialization heuristic — replacing drifting prompts with noisy averages of well-performing ones — stabilizes training. Prompt learning takes about 3 minutes per class.

Results

The main comparison covers ViT-S/B, DeiT-S/B, and Swin-S/B at W1.58A8, W3A3, W4A4, and W6A6, using either 10,000 real ImageNet images or 100,000 synthetic images:

Setting	Backbone	Best prior PTQ	Ours (Real)	Ours (Synth)
W1.58A8	ViT-S	4.84	68.45	63.71
W1.58A8	DeiT-B	57.75	78.07	77.18
W3A3	ViT-S	64.09	71.89	68.46
W4A4	ViT-S	76.68	78.35	77.61
W6A6	ViT-B	84.82	84.80	84.84

Several findings stand out. At W1.58A8, RepQ-ViT collapses entirely (below 0.2% everywhere) and FIMA-Q degrades severely, whereas the proposed method retains 63–79% top-1 accuracy — a result the authors state is, to their knowledge, the first successful sub-2-bit PTQ for Vision Transformers. This claim rests on the joint optimization enabling cross-block error compensation that sequential block reconstruction cannot provide. At W3A3 and W4A4 the method leads on most backbone/bit combinations, though FIMA-Q remains competitive or superior on Swin models at moderate bit-widths, which the authors attribute to Swin's local attention structure already stabilizing quantization. Synthetic calibration costs typically only 1–2% relative to real data, supporting the central data-free claim.

For the prompt-learning component, learned multi-mode prompts outperform raw text prompts across all bit-widths, with the largest gain of +3.6% at W1.58A8 on ViT-Small. Qualitative analysis shows raw prompts collapsing onto a single interpretation for ambiguous classes (e.g., kite as toy rather than bird), while t-SNE visualizations confirm that learned prompts yield feature distributions that spread more broadly and cluster closer to the real-image manifold than template-generated samples.

Limitations and open questions

The paper concedes several points. Gains on Swin models diminish at moderate and high bit-widths, indicating the joint-optimization advantage is architecture-dependent. The saturation of accuracy beyond 10,000 calibration samples bounds the benefit of additional data. The data-free pipeline depends on SD-Turbo's coverage of ImageNet concepts: prompts are supervised by a pretrained full-precision ViT classifier, so generation quality inherits any biases of that classifier, and the paper does not evaluate how the approach transfers beyond ImageNet-1K or to detection/segmentation tasks. The prompt-drift reinitialization is described as a heuristic without ablation of its sensitivity. Finally, the W1.58A8 configuration still uses 8-bit activations, leaving fully ternary weight-and-activation PTQ unaddressed.

Conclusion

This paper replaces sequential block-wise PTQ with a single end-to-end distillation-driven optimization over all quantization parameters, learnable channel rescaling, and weight refinement, requiring no labels and completing within hours on one GPU. Combined with a multi-mode prompt-learning scheme that synthesizes diverse, class-consistent calibration imagery from Stable Diffusion Turbo, it delivers state-of-the-art low-bit ViT quantization and extends PTQ into the ternary-weight regime previously reachable only via QAT. The principal open questions concern generalization beyond ImageNet classification, performance on architectures where local attention already mitigates quantization difficulty, and closing the residual gap between synthetic and real-data calibration at the lowest bit-widths.