---
title: Quantized MeZO (QZO) Optimization
url: https://www.emergentmind.com/topics/quantized-mezo-qzo
type: topic
---

# Quantized MeZO (QZO) Optimization

Quantized MeZO (QZO) refers to a class of optimization techniques that combine quantized neural network representations with zeroth-order (gradient-free) optimization strategies for the purposes of memory- and compute-efficient fine-tuning. QZO arises in the context of training and adapting large neural models—including language models—using low-bit (4–8 bit) numerical formats and forward-only loss queries, thus avoiding both backpropagation and full-precision storage. This approach addresses the significant memory and computational barriers posed by large-scale architectures when deployed in resource-constrained environments [2502.12346] [2505.13430].

## 1. Fundamental Problem Formulation

Quantized neural networks map real-valued parameters $\theta \in \mathbb{R}^d$ to low-precision representations through quantization operators. For a fixed quantization codebook or scalar scale $\Delta$, the quantized weight vector is
\[
\bar{\theta}_i = \operatorname{round}(\theta_i / \Delta_i), \qquad \theta_i = \Delta_i\,\bar{\theta}_i,
\]
with $\bar{\theta} \in \mathbb{Z}^d$ discrete codes. Fine-tuning in this regime involves minimizing the loss $\mathcal{L}(\theta; \mathcal{B})$, typically over a set of frozen codes and continuous scales:
\[
\min_{\Delta} \; \mathcal{L}(\Delta \odot \bar{\theta}; \mathcal{B}),
\]
or, in the fully quantized scenario, keeping both parameters and perturbations in low-bit format and performing optimization steps without gradient information or backpropagation [2502.12346] [2505.13430].

## 2. Quantized Zeroth-Order Gradient Estimation

Traditional zeroth-order (ZO) stochastic gradient descent estimates the gradient via finite differences along randomly sampled directions:
\[
g(w) = \frac{1}{n} \sum_{i=1}^n \frac{\mathcal{L}(w+\epsilon u_i) - \mathcal{L}(w-\epsilon u_i)}{2\epsilon} u_i,
\]
where $u_i \sim \mathcal{N}(0, I)$. Directly quantizing $u_i$ damages their distributional properties, introducing significant bias to gradient estimates. To circumvent this, QZO frameworks employ independent stochastic quantization for each perturbation: $u_{i,1} = Q_1(u_i)$, $u_{i,2} = Q_2(u_i)$, with stochastic rounding schemes ensuring $\mathbb{E}[u_{i,1}] = \mathbb{E}[u_{i,2}] = u_i$ and $\mathbb{E}[u_{i,1}u_{i,2}^T]=u_i u_i^T$. The unbiased quantized finite-difference estimator becomes
\[
\widehat{\nabla} \mathcal{L}(\bar{w}) = \frac{1}{n} \sum_{i=1}^n \frac{\mathcal{L}(\bar{w}+\epsilon u_{i,1}) - \mathcal{L}(\bar{w}-\epsilon u_{i,1})}{2\epsilon} u_{i,2}.
\]
This estimator, denoted Q-RGE2, retains unbiasedness even in ultra-low-bit formats (e.g., INT4), as opposed to naïve quantization (Q-RGE1), which exhibits rapidly increasing bias and instability [2502.12346].

## 3. Algorithmic Procedures and Implementation

The QZO update is performed entirely in low precision, with no backward pass required:
- For each minibatch, $n$ random perturbations $u_i$ are sampled and stochastically quantized to $(u_{i,1}, u_{i,2})$.
- The loss is queried at $\bar{w} \pm \epsilon u_{i,1}$, and the sensitivity $\mu_i$ is calculated.
- Model parameters are updated as
\[
\bar{w}_{t+1} = \bar{w}_t - \sum_{i=1}^n Q\left(\frac{\eta_t \mu_i}{n} u_{i,2}\right),
\]
where $Q[\cdot]$ denotes stochastic rounding and $\eta_t$ is the learning rate [2502.12346].

A variant, developed specifically for scenarios where only the quantization scales are optimized (with codes $\bar{\theta}$ fixed), estimates gradients w.r.t. $\Delta$ via forward-only perturbations:
- Two loss queries at $(\Delta+\epsilon z) \odot \bar{\theta}$ and $(\Delta-\epsilon z) \odot \bar{\theta}$ are performed for a random direction $z$.
- The directional derivative $d = (\mathcal{L}_+ - \mathcal{L}_-)/(2\epsilon)$ is obtained and clipped (to reduce variance), followed by a scale update:
\[
\Delta_i \leftarrow \max(\Delta_i - \eta_t d' z_i, 0)
\]
[2505.13430].

## 4. Quantization Schemes and Theoretical Properties

- **Supported datatypes:** INT4, INT8, and mixed low-precision formats (FP8, per-channel INT/Floating).
- **Optimization properties:** QZO with two-stage stochastic quantization yields provably unbiased gradient estimates in expectation, even in the presence of severe quantization [2502.12346].
- **Stability mechanisms:** Directional derivative clipping constrains variance without introducing bias, ensuring convergence and preventing training collapse in practical deployments [2505.13430].
- **Integration:** QZO can be layered atop any post-training quantization pipeline, such as GPTQ, AWQ, AQLM, and QLoRA. The only requirement is a continuous quantization scale or codebook [2505.13430].

## 5. Empirical Results and Efficiency Analysis

Extensive empirical studies demonstrate:
- **Memory footprint reductions:** Up to $2.94\times$ (full tuning) and $5.47\times$ (LoRA-tuning) reduction compared to quantized first-order methods. Memory cost is driven down to $3.6$ GB for 4-bit 7B-parameter models, enabling single-GPU training and adaptation [2505.13430] [2502.12346].
- **Performance parity:** On language model tuning tasks (GLUE, SQuAD, MultiRC), QZO achieves accuracy close to or surpassing full-precision MeZO and outperforms quantized first-order optimizers, especially under extreme (INT4/INT8) quantization [2502.12346].
- **Forward-only training:** No backpropagation or full-precision accumulators are needed. Only two forward loss queries per update, with highly efficient inference kernels on quantized hardware [2502.12346] [2505.13430].
- **Vision and diffusion tasks:** QZO successfully fine-tunes Stable Diffusion 3.5 Large (quantized 4-bit) using $12.4$ GB VRAM, far below the $86$ GB required for fp16 backpropagation. Similar qualitative benefits observed, though diffusion models display sensitivity to ZO noise [2505.13430].

## 6. Limitations and Extensions

- **Quantization error:** The quality of gradient estimation is constrained by quantization noise. Poor quantization of model weights amplifies stochastic estimation error.
- **Performance gaps:** There remains a gap to full-precision, backprop-based fine-tuning, especially for highly nonlinear objectives or ultra-low-bit settings.
- **Specialized perturbation:** For generative models (e.g., diffusion), ZO perturbations may interfere with scheduled denoising, impacting fidelity [2505.13430].

Proposed extensions include joint optimization of quantization scales and select full-precision parameters, adaptively scheduled perturbations, and advanced variance-reduced ZO estimators (antithetic sampling, multi-point queries) [2505.13430].

## 7. Historical and Nomenclatural Notes

Although “Quantized MeZO” (QZO or QuZO) was independently introduced in at least two contemporary works—“QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models” [2502.12346] and “Fine-tuning Quantized Neural Networks with Zeroth-order Optimization” [2505.13430]—both present the same core synthesis of quantized inference and zeroth-order optimization. The algorithms are orthogonal to specific quantization schemas and consistent with the MeZO (Memory-efficient Zeroth-order Optimization) principle—generalizing its applicability from full-precision to ultra-low-bit neural network fine-tuning.

A different but unrelated combinatorial quantity, termed “quantized Mező” numbers, appears in the context of $q$-analogs of Stirling and Bell identities, as in [1412.1110]; this usage is distinct and not connected to neural optimization.

Source: https://www.emergentmind.com/topics/quantized-mezo-qzo