Zero-Shot Quantization via Weight-Space Arithmetic

Abstract: We show that robustness to post-training quantization (PTQ) is a transferable direction in weight space. We call this direction the quantization vector: extracted from a donor task by simple weight-space arithmetic, it can be used to patch a receiver model and improve robustness to PTQ-induced noise by as much as 60%, without receiver-side quantization-aware training (QAT). Because the method requires no receiver training data, it provides a zero-shot, low-cost alternative to QAT for extremely low-bit deployment. We demonstrate this on Vision Transformer (ViT) models. More broadly, our results suggest that quantization robustness is not merely a byproduct of task-specific training, but a reusable feature of weight-space geometry that can be transferred rather than retrained.

• The paper introduces the Quantization Vector (QV) to transfer QAT-induced robustness via weight-space arithmetic.
• The methodology enables zero-shot improvement in ViT models under 3-bit quantization, achieving up to 60% gain in Top-1 accuracy.
• Results reveal that QAT-induced robustness is a transferable geometric property, reducing retraining costs for low-bit quantization deployment.

Zero-Shot Quantization via Weight-Space Arithmetic: Technical Analysis

Motivation and Background

The paper addresses robustness to extreme low-bit quantization in deep neural networks, specifically Vision Transformers (ViTs). Conventional post-training quantization (PTQ) is attractive for its data-free deployment but suffers drastic accuracy degradation below 4 bits, especially in ViTs due to their irregular activation distributions. Quantization-aware training (QAT) remedies this by integrating quantization constraints into the optimization, producing models significantly more robust at low precision but at considerable additional computational cost. The central research question is whether QAT-induced robustness constitutes a feature of parameter-space geometry that can be extracted from one task and transferred to another, circumventing the costly retraining required by QAT.

Methodology

The principal contribution is the introduction of the Quantization Vector (QV): the difference in parameter space between a standard fine-tuned checkpoint and its QAT counterpart for a given task. Formally, for a donor dataset $\mathcal{D}$ and a shared pre-trained initialization $\theta_{\text{pre}}$, the QV is defined as $$\rho_{\mathcal{D}} = \theta_{\mathcal{D},\text{QAT}} - \theta_{\mathcal{D}}$$. This vector is hypothesized to embody the "structural" robustness to quantization noise learned during QAT.

Zero-shot patching is achieved by applying this QV to an unrelated receiver model, $\theta_\mathcal{R}$, trained without QAT, according to $$\theta_{\mathcal{R} \leftarrow \mathcal{D}} = \theta_{\mathcal{R}} + \lambda \rho_{\mathcal{D}}$$, with $\lambda$ as a tunable scaling parameter. If the receiver shares the network architecture and initialization as the donor, linear operations in weight space are justified, leveraging the compatible loss basins established in prior work on weight-space arithmetic [ilharco2022editing], [wortsman2022model].

Evaluation is conducted by quantizing both the original and patched receiver models and measuring the Top-1 accuracy gap, $\Delta(\mathcal{D},\mathcal{R}) = \mathrm{Acc}(\mathrm{FQ}(\theta_{\mathcal{R} \leftarrow \mathcal{D})) - \mathrm{Acc}(\mathrm{FQ}(\theta_{\mathcal{R}))$, across dozens of image classification tasks and ViT architectures (ViT-T/S/B/L).

Empirical Results

Evidence demonstrates that patching ViT receivers with donor QVs yields up to 60% improvement in Top-1 accuracy under 3-bit symmetric per-channel PTQ, relative to unpatched models. Critically, even when donor and receiver tasks diverge substantially, the patched receiver consistently outperforms vanilla PTQ. The directionality of the QV is broadly transferable; modulation of $\lambda$ eliminates nearly all destructive interference, confirming that negative transfer at unit scale is largely attributable to step-size mismatch rather than invalidity of the robustness direction.

Additional key findings include:

• Universality of QV Direction: When the magnitude $\lambda$ is optimized per donor-receiver pair, almost all pairs experience positive transfer, and negative transfer is virtually eliminated, particularly for large-scale donors (e.g., ImageNet, Tiny ImageNet).
• Task Dependence: While most receivers benefit from robust donors, transferability varies with the donor's task; certain datasets induce QVs that generalize exceptionally well.
• Model Scale: The phenomenon is robust across ViT model sizes, though transfer magnitude and sensitivity to $\lambda$ scale with model capacity.

Baseline analysis confirms the severity of the low-bit regime: PTQ at 3-bit causes sharp drops in accuracy, with QAT recovering much of the lost performance. The QV patch bridges a significant portion of the PTQ-QAT gap without additional training on the receiver task.

Theoretical and Practical Implications

The finding that QAT-induced robustness exists as a transferable weight-space direction challenges the prevailing view that extreme quantization robustness must be "relearned" for each deployment scenario. Instead, it suggests that this robustness is a geometric property, reusable across tasks within the same architecture and initialization scheme.

Practically, this methodology enables zero-shot, data-free improvement of quantization robustness for arbitrary downstream ViTs. It substantially reduces the deployment cost for edge scenarios, federated learning, and large-scale multi-task systems where retraining is infeasible. The QV protocol can be further integrated with more advanced PTQ heuristics, possibly offering additional robustness gains.

Theoretically, the results reinforce recent work suggesting the loss landscape's geometry is central to quantization resilience [catalan2025training], [tabesh2025cage]. QAT moves solutions toward flatter minima, yielding robustness that is embodied in a compact vector in parameter space. The demonstrated efficacy of weight-space arithmetic for structural rather than just semantic properties (cf. task vector arithmetic [ilharco2022editing]) broadens the applicability of model-editing paradigms.

Future Directions

Open questions include:

• Extension to activation/attention quantization and mixed-precision regimes.
• Selection of optimal donors and magnitude scaling for unseen tasks without test-labels, possibly via calibration or meta-learning.
• Integration with more sophisticated PTQ preprocessing and quantizer adaptation strategies.
• Further analysis of curvature-aware QVs and multi-objective transfer, as well as interactions with model merging, continual learning, and robustness to other forms of noise.

Conclusion

The paper establishes that robustness to extreme post-training quantization can be encapsulated and transferred as a weight-space vector, unlocking efficient, zero-shot quantization robustness for ViTs across diverse tasks. The methodology shifts the quantization paradigm from task-specific retraining to geometric transfer, with both practical and foundational implications for scalable deployment of deep models at ultra-low precision.