AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Abstract: Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory. Mixed-precision quantization can substantially reduce this footprint by assigning different bit-widths to different experts. Existing approaches, however, typically rely on calibration data to estimate expert importance and determine bit allocation. For frontier MoE LLMs, the original training data, and hence the true training distribution, is proprietary and inaccessible. As a result, calibration sets are inevitably imperfect surrogates, and this can misestimate expert utilization and lead to suboptimal bit allocation. Motivated by the substantial cross-expert quality variability observed in modern MoE models, and by the success of Heavy-Tailed Self-Regularization (HT-SR) theory at predicting neural network model quality without access to training or testing data, we propose AlphaQ, a calibration-free bit-allocation method for MoE quantization. AlphaQ draws on HT-SR theory and follows a simple principle: experts with more heavy-tailed weight spectra are typically better trained and hence should receive higher bit-widths, while experts with weaker heavy-tailed structure can be quantized more aggressively. AlphaQ operationalizes this principle by measuring expert-wise spectral heavy-tailedness and solving a budget-constrained optimization problem that minimizes total quantization error under a global bit-budget constraint. Across several MoE models, AlphaQ consistently outperforms calibration-based baselines under matched bit budgets. Notably, on Qwen1.5-MoE, AlphaQ achieves near full-precision accuracy with an average expert precision of only 3.5 bits, while delivering more than 4$\times$ memory compression. Our code is available at https://github.com/Superone77/AlphaQ.

• The paper introduces a calibration-free approach for bit allocation in MoE quantization by leveraging spectral heavy-tailed metrics to assess layer importance.
• It employs PL_Alpha_Hill metrics and an ILP framework to determine fine-grained, layer-wise bit precision, achieving significant improvements in memory usage and accuracy.
• Empirical evaluations on multiple MoE-LLMs demonstrate up to 7.2× compression and robust cross-domain generalization without reliance on calibration data.

AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Introduction and Motivation

Mixture-of-Experts (MoE) architectures have become a central paradigm for scaling LLMs owing to their efficiency via sparse expert activation. Despite significant advances in post-training quantization (PTQ) for standard dense LLMs, deploying MoE-LLMs at scale remains challenging: even though only a sparse subset of experts is activated per token, all expert weights must be retained in device memory at inference, imposing severe memory bottlenecks. To mitigate this, mixed-precision quantization—assigning heterogeneous bit-widths across different parts of the model—has been explored. However, extant bit allocation schemes are fundamentally data-driven: they require calibration data to gauge expert importance or sensitivity, leading to biased importance estimates and poor generalization, especially when the calibration data does not faithfully represent the true deployment distribution.

The "AlphaQ" framework introduces a data-independent, calibration-free approach to bit allocation for MoE quantization, drawing on insights from Heavy-Tailed Self-Regularization (HT-SR) theory. Instead of relying on imperfect proxy data, AlphaQ exploits the spectral characteristics of expert weight matrices—quantified by PL_Alpha_Hill metrics—to measure training sufficiency and structural importance. This enables a principled mapping from real model weights to optimal, globally-constrained, layer-wise bit allocations, circumventing calibration bias and improving accuracy/memory trade-offs across architectures and deployment domains.

Domain Bias and Calibration Dependence in MoE Quantization

Existing data-driven bit allocation methods estimate expert importance by measuring expert activation and quantization error statistics on hand-selected calibration datasets. However, calibration data selection induces strong domain bias: models optimized for math, code, or general language datasets exhibit skewed bit-width assignments, overfit to the calibration domain, and degrade sharply on out-of-domain tasks.

Figure 1: Domain bias introduced by data-driven bit-width allocation in Mixtral-8×7B. Calibration on different domains yields distinct bit-width assignments and performance bias, illustrating reduced cross-domain generalization.

In "Mixtral-8×7B," allocation using math-oriented calibration data yields high precision for math-sensitive experts but impaired performance on code or commonsense tasks, and vice versa for code-calibrated models. Efforts to construct multi-domain or expert-balanced calibration sets address this only partially—the domain coupling remains intrinsic to the data-dependent pipeline. Thus, a calibration-free, data-independent allocation principle is essential for robust, generalizable quantization.

AlphaQ: Theory and Methodology

Spectral Heavy-Tailedness as an Importance Signal

AlphaQ leverages the HT-SR framework: in well-trained neural networks, important layers exhibit weight matrices with heavy-tailed empirical spectral density (ESD). Specifically, the degree of heavy-tailedness (smaller fitted power-law exponents, denoted by PL_Alpha_Hill) correlates with layer importance. Thus, experts/layers with lower PL_Alpha_Hill (heavier tails) are inferred to be more structurally important and should be quantized at higher precision, while lighter-tailed, less informative layers can be compressed more aggressively without inducing excessive accuracy loss.

AlphaQ computes the PL_Alpha_Hill metric directly from (full-precision) model weights, applying the Fixed-Aspect-Ratio Matrix Subsampling (FARMS) heuristic for consistent cross-module comparison. This metric is calibration-free, globally comparable, and robust across diverse model architectures and sizes.

Figure 2: Distribution of PL_Alpha_Hill across up/gate/down projections in three MoE-LLMs, revealing substantial variance both across and within blocks and experts.

Figure 3: Layer-wise PL_Alpha_Hill in sampled MoE blocks; up, gate, and down projections frequently exhibit disparate importance, motivating fine-grained layer-wise allocation.

Figure 4: Expert-wise PL_Alpha_Hill within a single MoE block, highlighting strong intra-block heterogeneity even within a single MoE layer.

Notably, the variance of PL_Alpha_Hill increases for modern, fine-grained MoEs (e.g., DeepSeekV2-Lite and Qwen1.5-MoE) versus vanilla MoEs (e.g., Mixtral-8×7B), emphasizing the necessity of high-resolution, global allocations rather than coarse, block- or expert-uniform strategies.

Budget-Constrained, Importance-Scaled Bit Allocation

AlphaQ formalizes bit allocation as an integer linear programming (ILP) optimization. The core objective is to minimize total expected quantization-induced degradation, modeled as the product of quantization noise (which falls exponentially with increasing bit-width) and an importance score (PL_Alpha_Hill-based, amplified by a data-independent curvature parameter $\gamma$). Thus, the per-layer allocation cost is

$$\eta_{l, b} = \left( \frac{\tilde{\alpha}}{\alpha_l} \right)^\gamma \cdot \mathrm{Var}(\mathbf{W}_l) 2^{-2b}$$

where $b$ is the candidate bit-width, $\tilde{\alpha}$ is the median PL_Alpha_Hill, and $\mathbf{W}_l$ is the weight matrix.

Global memory constraints—expressed in total available bits/quota—are encoded as hard constraints, and modern ILP solvers yield optimal configurations.

Figure 5: Comparison between AlphaQ (calibration-free) and standard data-driven quantization pipelines.

Furthermore, empirical analysis shows that neither quantization noise nor structural importance alone suffices: only the combined, importance-scaled metric provides a monotonic predictor of quantization-induced accuracy loss.

Figure 6: Joint relationship among PL_Alpha_Hill, quantization noise, and induced degradation; clear degradation emerges when high noise co-occurs with low PL_Alpha_Hill.

Empirical Evaluation

AlphaQ is evaluated on multiple state-of-the-art MoE-LLMs: DeepSeekV2-Lite, Qwen1.5-MoE, Mixtral-8×7B, and Qwen3-30B-A3B. The quantization is weight-only, groupwise, with a uniform 4-bit quantization for non-expert layers and bit allocation over $\{1,2,3,4\}$ bits for expert layers. Performance is measured via perplexity and zero-shot accuracy benchmarks.

Strong numerical results demonstrate AlphaQ's superiority:

• On Qwen1.5-MoE, AlphaQ achieves nearly the same average accuracy as the full-precision BF16 model with only 3.5 bits/layer, yielding over $4\times$ compression.
• For aggressive 2–2.5 bits/layer settings, AlphaQ outperforms both uniform and calibration-based PMQ, with less than $5\%$ relative drop in task accuracy.
• On OLMoE-1B-7B, AlphaQ outperforms PMQ even when multi-domain calibration is used, indicating superior cross-domain generalization.
• Ablation confirms that global budget allocation and layer-wise (vs. expert- or block-wise) granularity yield measurable gains in perplexity and accuracy.

Figure 7: Accuracy–speedup Pareto frontier on Mixtral-8×7B; AlphaQ consistently offers superior trade-offs over other schemes.

Efficiency, Implementation, and Overhead

AlphaQ not only improves accuracy-per-bit but also enables practical memory and compute scaling:

• On Mixtral-8×7B, memory usage drops from 96.8 GB (BF16) to 13.4 GB (2-bit quantized), achieving $7.2\times$ compression; speedups of $1.68\times$ are reported.
• On Qwen1.5-MoE, AlphaQ attains BF16-level accuracy using only 7.6 GB, $$\eta_{l, b} = \left( \frac{\tilde{\alpha}}{\alpha_l} \right)^\gamma \cdot \mathrm{Var}(\mathbf{W}_l) 2^{-2b}$$0 of the baseline footprint.

The offline PL_Alpha_Hill computation adds minimal preprocessing overhead—minutes for billion-scale models, a one-time expense amortized over all inference.

Bit allocations for different models under 2-bit budgets illustrate the fine-grained, adaptive nature of AlphaQ assignments:

Figure 8: Bit allocation of DeepSeekV2-Lite under a 2-bit budget.

Figure 9: Bit allocation of Qwen1.5-MoE under a 2-bit budget.

Figure 10: Bit allocation of Mixtral-8×7B under a 2-bit budget.

Theoretical and Practical Implications

AlphaQ marks a shift toward theory-grounded, parameter-centric model compression. By linking quantization robustness to spectral hypotheses from HT-SR theory, it provides:

• A practical framework for memory-efficient deployment of frontier MoE-LLMs without reliance on unavailable or biased calibration data.
• Strong empirical evidence that spectral heavy-tailedness is a reliable proxy for training sufficiency, extensible to other model compression and sparsity regimes.
• Improved domain generalization and reduced risk of task overfitting or underfitting due to calibration artifacts.

The potential for AlphaQ to be integrated with activation quantization, encoder-decoder/vision MoE models, or even adaptive test-time model reconfiguration constitutes promising future directions. Furthermore, the community may benefit from extending spectral diagnostics (PL_Alpha_Hill, FARMS, etc.) to broader model analysis, selection, and pruning workflows.

Conclusion

AlphaQ is a calibration-free, theoretically-grounded method for optimal bit allocation in MoE quantization. By exploiting parameter-derived importance via spectral heavy-tailedness, AlphaQ supersedes data-driven allocation schemes in accuracy, memory efficiency, and domain robustness. Empirical results over multiple MoE-LLMs underline its versatility and practical benefit, while its conceptual underpinning in HT-SR theory paves the way for future developments in compression, pruning, and deployment of large sparse models.