---
title: Trained Quantization and Weight Sharing
url: https://www.emergentmind.com/topics/trained-quantization-and-weight-sharing
type: topic
---

# Trained Quantization and Weight Sharing

Trained quantization and weight sharing are strategies designed to compress neural networks by reducing the number of distinct weight values stored and enforcing parameter sharing, often with minimal or no loss in predictive accuracy. These techniques have become fundamental for efficient deployment of deep models on resource-constrained systems, especially as model sizes have scaled to billions of parameters. This entry provides a technical overview of methodology, optimization schemes, representations, theoretical underpinnings, and empirical effects, emphasizing both classical and recent research advances.

## 1. Mathematical Formulations and Core Algorithms

Trained quantization and weight sharing operate by mapping dense, full-precision parameter tensors $W$ to a discrete set of shared weight values, referred to as the "codebook" $C = (c_1, ..., c_k)$, through an assignment mapping $\pi: \{1, \ldots, n\} \to \{1, ..., k\}$. The transformed weight for index $i$ becomes $\widetilde W_i = c_{\pi(i)}$ [1510.00149][2108.12704].

The learning objective follows a standard empirical risk minimization but is subject to a quantization constraint:
$$
\min_{C, \pi} \mathcal{L}(\widetilde W(C, \pi)),
$$
where $\mathcal{L}$ is the task loss. The optimal $(C, \pi)$ are typically chosen to minimize the within-cluster sum of squares, yielding a $k$-means quantization:
$$
\min_{C, \pi} \sum_{i=1}^n \|w_i - c_{\pi(i)}\|^2.
$$
Once the assignments $\pi$ are fixed, the codebook centroids $C$ can be fine-tuned, e.g., by minimizing downstream supervised loss via SGD or retraining the network with quantized weights.

Variants such as uniformly-binned quantization define $k$ uniformly spaced codebook entries and assign each weight to its nearest bin, reducing per-step complexity from $\mathcal{O}(Nkt)$ (k-means) to $\mathcal{O}(N)$ [2501.03095]. Ternary quantization restricts the codebook to $\{-S, 0, +S\}$ per layer, with thresholds and scaling factors trained jointly [1810.01018].

Recent Bayesian and stochastic weight sharing frameworks represent each weight $w_i$ as a random variable, typically Gaussian, and assign codebook entries via probabilistic or variational relaxation, facilitating uncertainty-aware quantization and more flexible cluster assignment [2309.13575][2505.17856].

## 2. Compression, Parameterization, and Storage Analysis

Compression is achieved through three primary mechanisms [1510.00149][2108.12704]:

- **Parameter sharing:** Each weight is replaced by an index into a codebook of $k \ll n$ unique values, yielding a storage cost of $n \cdot \lceil \log_2 k \rceil$ bits for assignments and $32k$ bits for the full-precision codebook.
- **Sparse encoding:** Pruning removes redundant weights, storing only non-zero entries.
- **Entropy coding:** Non-uniform index or codebook distributions are further compressed via Huffman or similar schemes, often achieving 20–30% additional storage reduction.

Empirically, deep compression pipelines utilizing pruning, trained quantization, and entropy coding have demonstrated compression ratios up to 49× (AlexNet from 240MB to 6.9MB; VGG-16 from 552MB to 11.3MB) with negligible or zero accuracy loss [1510.00149]. For fully connected layers, compression rates can reach 150× through strong pruning and weight sharing [2108.12704]. Evolutionary search strategies using uniform bin quantization and iterative bin merging have reported compression rates up to 15× on CIFAR-10 and 8.6× on ImageNet [2501.03095].

A summary of storage and compression:

| Method/Model               | Compression Rate | Top-1 Accuracy Drop | Codebook Size |
|---------------------------|-----------------|---------------------|---------------|
| Deep Compression (AlexNet)| 35×             | None                | ≤256          |
| Deep Compression (VGG-16) | 49×             | <0.5%               | ≤256          |
| Uniform+Binning (CIFAR-10)| 14–15× (w/ Huff)| <0.8%               | O(10–100)     |
| Soft Weight Sharing       | 40–160× (LeNet) | <0.1%               | 16–64         |
| Bayesian Quant. (PWFN)    | 8–13× (ImageNet)| Up to +1.6%*        | 143–325       |

*PWFN can sometimes improve accuracy over baseline [2309.13575].

## 3. Training, Optimization, and Finetuning

Most frameworks employ a multi-phase optimization:

1. **Initialization:** Codebook entries (centroids) are initialized either by random selection (Forgy), linearly spaced values, or k-means clustering of pre-trained weight values [1510.00149][2108.12704].
2. **Assignment:** Each parameter is mapped to the nearest centroid (hard assignment), or, in Bayesian methods, softly via cluster responsibilities or Mahalanobis-style distance as a function of the current uncertainty [1702.04008][2309.13575].
3. **Retraining / Fine-tuning:** With assignments fixed (or updated infrequently), centroids are updated to minimize downstream loss, optionally with back-propagation in the quantized network; gradients w.r.t. centroids are clustered sums over assigned weights [1510.00149]. In soft weight sharing, codebook means, variances, and mixing weights may be optimized jointly with network weights [1702.04008].
4. **Iterative refinement:** Post-processing steps such as iterative centroid merging are applied to further reduce codebook size while tolerating bounded accuracy loss [2501.03095]. PWFN alternates partial fixing and retraining rounds guided by Bayesian uncertainty [2309.13575].

Mixed-precision and weight-coupling frameworks optimize a super-network under all bit-width configurations simultaneously, employing techniques such as interference-mitigating bit-width freezing and feature alignment to enable retraining-free exploration of Pareto-optimal quantization schedules [2401.01543].

## 4. Theoretical and Information-Theoretic Perspectives

Trained quantization and weight sharing are closely linked to minimum description length (MDL) principles—balancing the bits required to encode model parameters with the error term encoding performance on training data [1702.04008]. A typical loss objective in soft weight-sharing is:
$$
\mathcal{L}(w,C) = -\log p(T|X,w) - \tau \log p(w;\{\mu_j,\sigma_j,\pi_j\}),
$$
where the second term penalizes weight complexity via a learned mixture prior.

Bayesian and variational extensions treat quantization as a relaxation, where weights are continuous random variables that gradually collapse to discrete codebook centers, with weight uncertainty $\sigma_i$ dictating compressibility: weights in flat loss regions have larger $\sigma_i$ and are quantized more aggressively [2309.13575][2505.17856]. This connects quantization to sharpness-aware minimization, as wider posterior modes allow more aggressive sharing with limited impact on accuracy.

In the stochastic regime, weight distributions are clustered in a lower-dimensional space (e.g., $(\mu,\sigma)$ for Gaussian mean-field) and merged using Wasserstein barycenters, yielding compressible mixtures without compromising uncertainty quantification [2505.17856].

## 5. Codebook Structures, Parameter Sharing, and Advanced Techniques

Codebooks range from small sets of scalars ($k$ typically 16–256) to low-rank scaling matrices for large language models [2407.11534]. In extreme quantization, ternarized codebooks $\{-S,0,+S\}$ are trained via closed-form scaling using truncated Gaussian approximations, achieving state-of-the-art performance at sub-3% accuracy drop for full ImageNet [1810.01018]. 

Layer-wise or global codebook architectures can be employed, with layer-wise adaptation often boosting accuracy in heterogeneously distributed weights [1510.00149]. Mixed-precision and coupled-weight quantization methods enable post-training selection of per-layer bit-widths via fast inference-only search, with no retraining required after the initial shared-weights cycle [2401.01543].

Low-rank parametric codebooks (as in LRQ) scale the step-size per weight through a low-rank factorization $A = UV + r_\text{vec} + c_\text{vec}$, improving generalization and stability over full-rank alternatives while dramatically reducing learnable codebook parameters [2407.11534].

Probabilistic weight fixing and stochastic weight sharing enable iterative, uncertainty-aware codebook fixing and aggressive quantization, allowing state-of-the-art entropy (<3 bits/weight) and O(10^2) unique values even in large transformer architectures [2309.13575][2505.17856].

## 6. Empirical Effects, Benefits, and Limitations

These methods have demonstrated that neural networks can be compressed 10–100× with minimal or even negative accuracy loss, validated across vision, language, and regression tasks [1510.00149][2108.12704][2501.03095][1702.04008]. For fully connected layers, compression up to 150× is achievable [2108.12704]. Bayesian and soft-clustering variants further improve noise resilience, uncertainty calibration, and generalization [2309.13575][2505.17856].

On large-scale benchmarks:

- AlexNet: Pruning $9\times$ followed by quantization ($8$ bits conv, $5$ bits fc) gives $27\times$ compression; with Huffman reaches $35\times$ with no accuracy loss [1510.00149].
- VGG-16: $49\times$ compression, $31.17\%$ Top-1 after compression vs $31.50\%$ baseline [1510.00149].
- Uniform binning + evolutionary merging yields $15\times$ on CIFAR-10 and $8.6\times$ on ImageNet, with $<0.8\%$ accuracy degradation [2501.03095].
- Soft weight sharing attains $40$–$162\times$ on MNIST/CIFAR with $<0.1\%$ accuracy loss [1702.04008].
- Stochastic and probabilistic frameworks achieve $50$–$100\times$ compression with ≤2\% accuracy loss and faithful uncertainty estimates [2505.17856][2309.13575].

Benefits include dramatically reduced storage and energy costs, inference acceleration, and model portability to constrained devices. A limitation is that uniform quantization may underutilize non-uniformly distributed parameter modes [2501.03095]; Bayesian and variationally relaxed schemes help to mitigate this. Some frameworks also note sensitivity in codebook initialization and regularization strength, particularly for soft and stochastic weight sharing [1702.04008][2309.13575].

## 7. Directions, Variants, and Ongoing Research

Trained quantization and weight sharing continue to evolve with advances in mixed-precision search [2401.01543], structure-agnostic multi-objective evolution [2501.03095], low-rank scaling for LLMs [2407.11534], Bayesian variational relaxations [2309.13575], and stochastic weight clustering coupled with principled uncertainty estimation [2505.17856]. 

Areas of ongoing investigation include:

- Joint optimization of codebook structure, entropy coding, and bit-allocation per layer or block.
- Extending quantization-aware training to handle uncertainty calibration (critical for applications in decision-critical systems).
- Combining sparse and quantized representations with advanced source coding for further compression [2108.12704].
- Adaptive, task-aware codebooks and structure-preserving quantization for efficient adaptation in continual and federated learning scenarios.

A plausible implication is that, as networks and pretraining datasets scale, weight sharing and trained quantization will remain critical in the design of efficient, deployable, and robust deep models across a wide variety of computational architectures and modalities.

Source: https://www.emergentmind.com/topics/trained-quantization-and-weight-sharing