Layer-wise and Non-uniform Quantization

• Layer-wise and non-uniform quantization is a technique that assigns varied precision across network layers to optimize model size, efficiency, and accuracy.
• It leverages sensitivity metrics like cosine dissimilarity, z-score distributions, and Fisher proxies to guide dynamic bit allocation.
• Empirical results demonstrate substantial gains in compression and efficiency for CNNs, transformers, and LLMs with minimal accuracy loss.

Layer-wise and non-uniform quantization refers to precision reduction schemes in deep neural networks where different layers—or, in the most granular cases, subcomponents—are assigned different quantization parameters such as bit-width or value mapping, with the goal of optimizing the tradeoff between model size, computational efficiency, and predictive accuracy. Unlike uniform quantization, which assigns a fixed bit-width and quantization profile globally, layer-wise and non-uniform schemes explicitly leverage the non-uniform sensitivity of different network components to quantization noise and the heterogeneous statistical structure of weights and activations. These methods have achieved substantial gains in compression rate and inference efficiency across CNNs, transformers, and LLMs, and underpin state-of-the-art deployment practices for both edge and server-class AI inference.

1. Principles and Rationale for Layer-wise and Non-uniform Quantization

Layer-wise and non-uniform quantization evolved from the observation that neural network layers—owing to their distinct roles and statistical profiles—react heterogeneously to quantization noise, and that model-wide uniform bit-width assignments are frequently suboptimal. Early studies highlighted that reducing precision in select layers, while maintaining higher precision elsewhere, can maintain model fidelity at drastically lower average bit-widths.

Formally, a layer-wise quantization regime introduces a bit-width allocation vector $\{b_\ell\}$ for layers $\ell=1\ldots L$, or an even finer $b_{i,j}$ for channels or filers, in contrast to global assignment. Non-uniform quantization further designates non-equally spaced quantization intervals or layer-specific codebooks, allowing the quantizer to focus representational power on statistically dense or more sensitive regions of the parameter or activation distributions.

The theoretical justification for such schemes traces to noise-propagation, robustness analysis, and (in the case of weights) the propagation of quantization error to final-layer logits and subsequent classification accuracy as a function of both local and extrinsic properties of each layer (Zhou et al., 2017, Gluska et al., 2020). Results show that bit assignment and quantization profile should be driven by explicit measures of per-layer sensitivity, typically derived from the distribution of weights, activations, or model performance under targeted perturbations (Dumitru et al., 2024, Zhang et al., 18 Mar 2026, Sun et al., 2022).

2. Quantitative Layer Importance Metrics and Allocation Algorithms

Designing an effective layer-wise quantization scheme requires robust, computationally feasible metrics to assess per-layer or per-module sensitivity to quantization. Several approaches have been introduced:

• Cosine Dissimilarity and Output Change (LIM): For transformer layers, the Layer Input Modification (LIM) metric quantifies representational change per layer as the negative cosine similarity between input embedding $x_\ell$ and output embedding $y_\ell$:

$$\mathrm{LIM}(L_\ell) = -\frac{x_\ell \cdot y_\ell}{\|x_\ell\|\,\|y_\ell\|}.$$

Higher-magnitude values indicate higher logical importance, warranting more bits (Dumitru et al., 2024).

• Z-score Distribution (Outlier Fraction, ZD): The fraction of weights with z-score exceeding one standard deviation in a layer reflects structural importance:

$$\mathrm{ZD}(L_\ell) = \frac{|\{j: z_{\ell j} > 1\}|}{N_\ell}, \quad z_{\ell j} = \frac{w_{\ell j} - \mu_\ell}{\sigma_\ell}.$$

Layers with higher ZD are prioritized for high precision (Dumitru et al., 2024).

• Dual Sensitivity (NSDS): Sensitivity is quantified using a combination of numerical vulnerability (excess kurtosis of weights) and structural expressiveness (role-aware spectral capacity), normalized robustly over layers via MAD-Sigmoid and aggregated with Soft-OR logic. Multi-part modules (e.g., attention/query and output/value matrices in transformers) are evaluated individually before aggregation (Zhang et al., 18 Mar 2026).
• Fisher-based or Hessian-trace Proxies: The Fisher information trace per layer, often scaled by layer type, offers a theoretically grounded metric for quantization-induced loss. This is used in integer programming-based assignment (Kim et al., 13 Nov 2025).

The typical allocation process involves:

1. Scoring all layers using one or more metrics.
2. Sorting layers in descending or ascending order of sensitivity.
3. Assigning available bits under a budget constraint (average bit-width or total model size).
4. Optionally, refining the assignment via iterative greedy swaps, integer programming, or Bayesian optimization for fine granularity or accuracy-memory trade-off (Dumitru et al., 2024, Kim et al., 13 Nov 2025, Nascimento et al., 2020).

3. Non-uniform Quantization Mappings and Codebook Design

Non-uniform quantization encompasses a variety of strategies:

• Power-based Quantizers (PowerQuant): Quantization is achieved by learning an exponent $a$ so that $x \mapsto \operatorname{sign}(x)|x|^a$ followed by affine rescaling and rounding matches the original distribution within a layer, enforcing operation-preserving (automorphism) properties for hardware and mathematical compatibility. The optimal $a$ minimizes the $\ell=1\ldots L$0 reconstruction error post round-trip quantization (Yvinec et al., 2023).
• Cluster-based Quantization (Lloyd/k-means, KNQ): Layer-wise k-means clustering determines centroids minimizing quantization error, assigning quantization levels to the centroid per weight/activation (Sun et al., 2016, GVSL et al., 2020).
• Error-balanced Superposition (AUSN): Weights are represented as a sum of power-of-two atoms, splitting bits between "basic" coverage (range) and "subdivision" (resolution), with the number of atoms adaptively set per layer to balance clipping and rounding error (Fangxin et al., 2020).
• Piecewise and Data-driven Non-uniform Binning: Clustering per-layer weights into subranges using outlier detection (IQR) and assigning bins in proportion to data density in each subrange reduces quantization MSE (GVSL et al., 2020).
• Row- or Channel-wise Learned Codebooks (GANQ): Lookup-table-based per-row quantization, where codebooks are adaptively optimized to minimize layer output error given input batch statistics, is hardware-aligned for efficient matrix multiplication on modern GPUs (Zhao et al., 22 Jan 2025).

The choice of mapping is determined by hardware constraints, statistical properties of the target layer, and whether the scheme is post-training, quantization-aware, or data-free.

4. Empirical Performance, Trade-offs, and Practical Guidelines

Multiple studies demonstrate that well-designed layer-wise and non-uniform quantization methods reliably outperform global or model-wise uniform baselines at the same average bit-width, delivering superior accuracy-compression trade-offs across domains:

• Transformer and LLMs: Allocating 2 bits to up to 25–50% of the least critical layers (by LIM or ZD) with the rest at 4 bits yields $\ell=1\ldots L$1 accuracy loss versus full-precision or all-4-bit baselines; random layer selection severely degrades performance (Dumitru et al., 2024). In LLMs with 13B+ parameters, models tolerate more aggressive precision drop than compact models (Dumitru et al., 2024, Zhang et al., 18 Mar 2026).
• CNNs and ViTs: Non-uniform allocation via search (MTGP, ILP, or ad hoc) saves $\ell=1\ldots L$2--$\ell=1\ldots L$3 memory with negligible (<0.5%) accuracy loss in standard vision architectures (Nascimento et al., 2020, Kim et al., 13 Nov 2025, Sun et al., 2016). Mixed-precision transformer quantization achieves state-of-the-art error rates, with fastest convergence and highest hardware efficiency for per-layer assignments (Kim et al., 13 Nov 2025).
• Filter- and channel-level granularity: Assigning bit-width per filter based on class-specific critical path scores can outperform even layer-wise schemes, allowing targeted pruning and fine-grained bit allocation without hardware modification (Sun et al., 2022).

Empirical practices include:

• Preferring meaningful layer ordering for low-bit assignment, e.g., using representational shift, kurtosis, or Fisher scores (Dumitru et al., 2024, Zhang et al., 18 Mar 2026, Kim et al., 13 Nov 2025).
• Using only two bit-widths per layer (e.g. 2 and 4 bits) for simplicity; a third rarely adds enough value to justify the complexity (Dumitru et al., 2024).
• Treating input/embedding and output projection layers conservatively, as these are typically the most sensitive (Nargund et al., 7 Feb 2026, Dumitru et al., 2024).
• Hybrid schemes (per-channel where needed) for layers where per-tensor quantization is insufficient, assessed via post-quantization divergence (GVSL et al., 2020).

5. Advanced Methodologies: Calibration-free and Data-independent Approaches

Recent algorithms attain calibration-free or data-independent layer-wise and non-uniform quantization, further broadening deployment options:

• Calibration-free sensitivity (NSDS): Sensitivity is computed via purely structural properties (weight statistics, spectral decomposition), with bit-allocation performed by robust aggregation, requiring no data or calibration set (Zhang et al., 18 Mar 2026).
• Data-free quantizer design (PowerQuant): Layer-wise exponents are tuned solely by weight reconstruction error, and the overall approach is compatible with integer-only hardware with minimal runtime impact (Yvinec et al., 2023).
• Synthetic calibration (Retro-Synthesis): Faux input batches are generated to match the per-layer statistics of a frozen FP model, supporting fully data-independent, post-training layer-wise quantization and non-uniform cluster-based schemes (GVSL et al., 2020).

These approaches have demonstrated comparable or superior accuracy on models with or without BatchNorm, and can efficiently scale to ImageNet/GLUE-scale benchmarks.

6. Hardware and Implementation Considerations

Layer-wise and non-uniform quantization interactions with hardware span several implementation axes:

• Complexity and Efficiency: Hardware-friendly choices, such as ENQ or Power-of-2 atom-based coding, can avoid costly lookup tables, multipliers, or decoders. LUT-based schemes (GANQ) can be mapped directly onto modern GPU tensor-cores for maximal speedup, but impose memory and bandwidth constraints (Fangxin et al., 2020, Zhao et al., 22 Jan 2025).
• Inference Efficiency: Correctly designed layer-wise schemes maintain tensor contiguity and memory alignment, enabling use of standard BLAS/backends; per-row or per-channel non-uniform codebooks demand careful balance between memory and speedup.
• Adaptive and Dynamic Strategies: Super-network training with per-layer variable quantizers enables data-dependent bit-width selection at inference (MDP-driven policy), achieving optimal BitOps per instance (Tang et al., 2022).
• Trade-off Tuning: Mixed-precision quantization introduces an explicit cost-accuracy surface that allows practitioners to target distinct deployment scenarios (tight memory, latency, etc.) with a single model (Kim et al., 13 Nov 2025).

7. Limitations and Future Directions

Despite substantial advances, several open questions and emerging directions persist:

• Fine-grained Allocation vs. Hardware Alignment: Increasing granularity (filter, neuron) can yield tighter accuracy but eventually breaks hardware mapping; most systems strike a balance at per-layer or per-row granularity (Sun et al., 2022).
• Sensitivity Metric Transferability: Metrics such as kurtosis or Fisher may underperform on completely novel model families; calibration- or data-driven hybridization remains an area of ongoing research (Zhang et al., 18 Mar 2026, Kim et al., 13 Nov 2025).
• Outlier Management: Some layers require dedicated outlier handling (clipping, per-channel scaling), which must be locally targeted to avoid introducing systematic bias (Gluska et al., 2020).
• Automated Search and Adaptive Inference: Layer-wise assignments increasingly leverage automated Bayesian or RL-based search for non-uniform bit allocations, including dynamic sample-dependent policies for instance-adaptive inference (Nascimento et al., 2020, Tang et al., 2022).

Layer-wise and non-uniform quantization is therefore a critical axis along which neural model compression and efficient inference advances, integrating quantitation from statistical physics, information theory, and numerical optimization to enable scalable and accurate AI deployment in both high-performance server and edge environments.