---
title: 'ZipNN: Model Merging & Lossless Compression'
url: https://www.emergentmind.com/topics/zipnn
type: topic
---

# ZipNN: Model Merging & Lossless Compression

ZipNN is an overloaded term in the arXiv literature. In one line of work, it denotes **Multi-Task Zipping (MTZ)**, a framework for **cross-model compression** that merges correlated, pre-trained neural networks by **layer-wise neuron sharing** and light retraining [1805.09791]. In a later and distinct line of work, it denotes a family of **lossless compression** methods for neural network tensors that exploit the statistical structure of floating-point representations—especially the skew of exponent fields—to reduce storage and transmission costs while preserving exact bitwise reconstruction [2411.05239]. Subsequent work extends this latter ZipNN formulation to **BF16 delta checkpoints**, **FP8 and FP4 formats**, and **K/V caches** used in large language models [2508.19263].

## 1. Dual usage of the term

The two established uses of “ZipNN” concern different objects and different compression levels.

| Usage | Object being zipped | Core mechanism |
|---|---|---|
| Multi-Task Zipping | Multiple pre-trained networks | Layer-wise neuron sharing |
| ZipNN lossless compression | Floating-point tensors and model artifacts | Exponent extraction, byte grouping, entropy coding |

In the MTZ sense, “zipping” means constructing a joint model \(M^C\) from separately trained models \(M^A, M^B, \dots\), with a shared subspace and task-specific subspaces. In the lossless-compression sense, “zipping” means representing model files, checkpoints, gradients, optimizer states, or K/V caches in a compact encoded form and then reconstructing the exact original bytes during decompression.

This suggests that the shared label is conceptual rather than methodological: one research line compresses **across models** by merging representations, whereas the other compresses **within stored tensor representations** by exploiting floating-point structure.

## 2. Multi-Task Zipping as cross-model compression

MTZ addresses a setting in which multiple correlated tasks are already served by separate deep neural networks, but storing and running all of them independently is prohibitive on mobile or edge devices [1805.09791]. Traditional pruning, quantization, and low-rank factorization reduce redundancy **within** a single network; MTZ instead targets redundancy **across** networks.

The central mechanism is **layer-wise neuron sharing**. For corresponding hidden layers \(l\) in models \(M^A\) and \(M^B\), MTZ identifies neurons that perform similar functions, merges selected pairs into shared neurons, and retains the remainder as task-specific neurons. If the original layers contain \(N^A_l\) and \(N^B_l\) neurons, then after zipping layer \(l\) the combined model contains \(\tilde N_l\) shared neurons, \(\hat N^A_l = N^A_l - \tilde N_l\) task-\(A\)-specific neurons, and \(\hat N^B_l = N^B_l - \tilde N_l\) task-\(B\)-specific neurons.

The resulting architecture is a **multi-head, partially shared backbone**. During inference for task \(A\), the shared neurons and the \(A\)-specific neurons are active, while \(B\)-specific connections are disabled; task \(B\) is handled symmetrically. The procedure is therefore not joint training from scratch with a manually chosen sharing pattern, but a post hoc transformation of pre-trained models into a single composite model.

A key practical motivation is that MTZ is designed to inherit information from each original model and then require only **light retraining** to re-boost task accuracy. In the VGG-16 experiments, the paper reports that MTZ can fully merge the hidden layers of two VGG-16 networks with a **3.18% increase in the test error averaged on ImageNet and CelebA**, or share **39.61% parameters** between the two networks with **<0.5% increase in the test errors for both tasks**. The number of iterations required to retrain the combined network is reported as **at least 17.8 times lower than that of training a single VGG-16 network**.

## 3. MTZ mathematical formulation and zipping procedure

MTZ formalizes neuron sharing through a layer-wise error criterion. For layer \(l\), it defines task-specific errors before and after zipping and combines them as
\[
E_l = \alpha E^A_l + (1-\alpha) E^B_l, \qquad \alpha \in (0,1).
\]
The change in error is approximated by a second-order Taylor expansion,
\[
\delta E \approx \frac{1}{2}\,\delta \boldsymbol{\theta}^{\top}\mathbf{H}\,\delta \boldsymbol{\theta},
\]
with the gradient term neglected at a local minimum. Following Dong et al. (L-OBS), the method uses **layer-wise Hessians** rather than a full-network Hessian [1805.09791].

For a candidate merge between neuron \(i\) in model \(A\) and neuron \(j\) in model \(B\), only the incoming weights from already shared neurons in layer \(l-1\) are updated. The layer-wise Hessians reduce to covariance matrices of layer inputs, scaled by \(\alpha\) and \(1-\alpha\). Enforcing equality of the post-update incoming weight vectors yields a constrained optimization problem whose solution defines both the optimal weight adjustments and the minimum induced error increase.

This leads directly to the MTZ **neuron functional difference metric**
\[
d[\tilde{\mathbf{w}}^A_{l,i}, \tilde{\mathbf{w}}^B_{l,j}] = \delta E_l^{\mathrm{opt}},
\]
where smaller \(d\) indicates that two neurons can be merged with smaller approximate increase in layer-wise error. The update rule
\[
f(\tilde{\mathbf{w}}^A_{l,i}, \tilde{\mathbf{w}}^B_{l,j})
\]
produces the shared incoming weight vector for the merged neuron.

The algorithm operates layer by layer. Its inputs are the pre-trained weight matrices, training data for both tasks, the balance coefficient \(\alpha\), and either a prescribed number of shared neurons \(\tilde N_l\) or a threshold \(\varepsilon_l\) on acceptable neuron-pair error. At each hidden layer, MTZ performs a forward pass to obtain layer inputs, computes layer-wise Hessians, evaluates all candidate pairwise functional differences, selects the pairs with smallest \(d\) or those satisfying \(d < \varepsilon_l\), updates the selected weights, permutes the corresponding rows in the next layer to preserve connectivity, and then performs a small number of SGD iterations to restore accuracy.

The paper interprets this as a **loss-aware matching and merging** procedure rather than weight averaging or heuristic alignment. The theoretical motivation is that minimizing local layer-wise error should keep the overall network deviation small, and the empirical results support that interpretation.

## 4. Empirical behavior of network zipping

The MTZ evaluation spans same-task model merging, cross-task model merging, and sequential zipping of multiple residual networks [1805.09791].

For **LeNet-300-100** and **LeNet-5** on MNIST, the goal was to fully share all hidden layers between differently trained models for the same task. After full hidden-layer zipping and light retraining, the reported errors were: **1.64%** for dense LeNet-300-100 after starting from **1.57%** and **1.60%**, **1.83%** for sparse LeNet-300-100 after starting from **1.80%** and **1.81%**, **0.93%** for dense LeNet-5 after starting from **0.89%** and **0.95%**, and **1.29%** for sparse LeNet-5 after starting from **1.27%** and **1.28%**. Retraining required **550**, **800**, **600**, and **1200** iterations respectively. The paper further reports that even **without retraining**, fully merging both hidden layers of LeNet-300-100 incurs only **~1.5% absolute increase in test error**, whereas **random neuron sharing yields ~33.47% error**.

For **two VGG-16 networks on different tasks**, task \(A\) was ImageNet ILSVRC-2012 classification with top-5 error **≈10.31%**, and task \(B\) was CelebA facial attribute classification with final error **≈8.50%**. In the **fully shared** setting, all **13 CONV layers + fc6 + fc7** were shared and only the final classification layers remained task-specific; the final errors were **14.07% top-5** on ImageNet and **11.09%** on CelebA. In the **adaptive sharing** setting constrained to **<0.5%** error increase for both tasks, the final errors were **10.75% vs 10.31% baseline** on ImageNet and **8.95% vs 8.50% baseline** on CelebA. The overall parameter sharing was **39.61%**, with **56.94% shared** in convolutional layers and **38.17% shared** in **fc6 & fc7**. Total zipping and retraining required **20,650 iterations**, reported as **≥ 17.9× fewer** than training one VGG-16 from scratch.

The paper also compares MTZ with a **fully shared multi-task VGG-16 trained from scratch** with the same topology and parameter count as the fully shared MTZ-zipped model. The from-scratch model reached **14.88%** on ImageNet and **13.29%** on CelebA, which was worse than MTZ under the same sharing topology. This is used to argue that zipping pre-trained models can outperform naïve joint training with a fixed sharing pattern.

For **five ResNet-28 networks** widened as in Zagoruyko and Komodakis, trained separately on **CIFAR-100, GTSR, Omniglot, SVHN, and UCF101**, MTZ sequentially zipped them with a target of **90% compression** among the five. The baseline of five separate models had mean error **≈17.95%** across tasks; the joint zipped model, with **≈1.5× the parameters of a single ResNet**, had mean error **≈18.20%**, corresponding to an **average accuracy drop ≈ 0.25% absolute**.

These experiments delimit MTZ’s operating regime. The method is most effective when tasks are sufficiently correlated that many neuron pairs exhibit small \(d\); when tasks are less related, fewer pairs satisfy the error criterion and the amount of safe sharing decreases.

## 5. ZipNN as lossless compression for AI models

In the later ZipNN line, the problem is no longer cross-model merging but **lossless compression of model representations** for storage and transmission [2411.05239]. The method targets dense tensors such as weights, biases, normalization parameters, embeddings, gradients, and optimizer states in common floating-point types including **BF16**, **FP16**, and **FP32**. Decompression restores the exact original tensor bytes, so the logical model and datatype remain unchanged.

The core observation is that floating-point tensors are not statistically homogeneous at the bit level. A value is represented as
\[
x = (-1)^s \cdot 2^{e-127} \cdot (1.m)
\]
for FP32, with analogous layouts for BF16 and FP16. In trained neural networks, exponent values occupy a narrow range: for **BF16/FP32 weights in language models and ResNet**, out of **256 possible exponent byte values**, only **≈40** appear at all, and the **top ≈12 exponent values** account for **≈99.9%** of all parameters. ZipNN therefore separates the exponent bytes from the remaining sign and mantissa bits.

The 2024 formulation organizes compression into four stages: **preprocessing**, **bit-level reorganization**, **per-group compression**, and **reconstruction**. Exponent extraction creates dedicated exponent streams; for “clean” models, ZipNN also applies **byte grouping** so that each byte position of the mantissa and sign is compressed separately. Compression is then performed per **chunk** and per **byte group**. The method favors **Huffman-only entropy coding** on exponent streams because LZ-style repetition removal contributes little on these distributions and can reduce effectiveness by adding back-pointer metadata. For zero-heavy streams, especially in deltas and some training artifacts, ZipNN can switch to **Zstd**. Chunk-level independence enables parallel compression and decompression and is recorded through a metadata map.

The paper distinguishes **regular** models from **clean** models. Regular models are typical trained models without heavy rounding or postprocessing; clean models have significant rounding or format conversion after training, leading to many zero or structured mantissa bits. This distinction materially affects compressibility. For regular BF16 models such as **Falcon-7B**, **Bloom**, **OpenLLaMA-3B**, **Mistral-7B**, and **Llama-3.1-8B-Instruct**, the reported compressed sizes cluster at **66.3%–67.4%** of original size. For regular FP32 models such as **wav2vec**, **BERT-base**, and **OLMo-1B**, the reported compressed sizes cluster around **83.0%–83.3%**. For clean FP32 models, the reported compressed sizes are substantially lower: **41.8%** for **XLM-RoBERTa**, **48.1%** for **CLIP**, and **33.7%** for **T5-base**.

The comparison with general-purpose compression is central. On a **1 GB slice** of **Llama-3.1-8B-Instruct (BF16)**, **Zstd** produced **77.7%** compressed size with compression and decompression speeds of **0.71 GB/s** and **1.02 GB/s**, whereas **ZipNN** produced **66.4%** compressed size with **1.15 GB/s** compression and **1.65 GB/s** decompression. On **OLMo-1B (FP32)**, ZipNN reduced size from **92.3%** under Zstd to **83.2%**, while increasing compression speed from **0.97 GB/s** to **1.64 GB/s** and decompression speed from **1.02 GB/s** to **2.48 GB/s**. On **XLM-RoBERTa (FP32, clean)**, ZipNN reduced size from **57.4%** under Zstd to **42.9%**, with compression speed increasing from **0.18 GB/s** to **0.83 GB/s** and decompression speed from **0.77 GB/s** to **1.41 GB/s**.

The paper’s broader claim is that ZipNN “often sav[es] **33%** and at times reduc[es] **over 50%** of the model size,” and that on popular models such as **Llama 3**, it achieves space savings **over 17% better than vanilla compression** while also improving compression and decompression speeds by **62%**.

## 6. Low-precision extensions: BF16 deltas, FP8, FP4, and K/V caches

The 2025 extension generalizes the lossless-compression ZipNN framework to lower-precision floating-point formats and runtime artifacts [2508.19263]. The same structural principle is retained: decompose each tensor into sign, exponent, and mantissa streams, then apply entropy coding—especially Huffman coding—to the exponent stream, whose empirical distribution remains skewed even in lower precision.

For **BF16 delta checkpoints**, the pipeline first computes
\[
D = C_{t+1} \oplus C_t
\]
between consecutive checkpoints, where XOR is applied bitwise over 16-bit BF16 words. This increases the number of zeros when training is stable or adjacent checkpoints are similar. The resulting sign, exponent, and mantissa streams are then compressed per checkpoint, per layer file, and in fixed-size chunks. On **llm360/Amber (6.74B parameters)**, the paper reports exponent-stream compression ratios as low as **0.07**, mantissa ratios around **0.69–0.92**, and total compressed delta checkpoint size down to **38%** of the original in later training stages.

For **FP8 weights**, especially **E4M3**, each value comprises **1 sign bit, 4 exponent bits, and 3 mantissa bits**. Despite the smaller exponent field, the exponent distribution remains compressible. The paper reports exponent-stream ratios in the **0.20–0.30** range, mantissa ratios around **0.80+**, and overall FP8 tensor ratios of **55–70%** of original size.

For **K/V caches** in BF16 and FP8, the paper recommends mostly **static Huffman codes** derived from representative cache samples, updating them only if compression degrades. Compression is performed per **layer** and often per head. The reported exponent compression ratios are **~0.25–0.45** for FP8 caches and **<0.20** for BF16 caches, with overall K/V cache memory reductions of **20–30%** and negligible impact on latency when static Huffman codes are used. The paper states that this is, to its knowledge, the **first demonstration of effective lossless compression of K/V caches**.

For **FP4 microscaling formats**, the outcome is different. The paper treats the 4-bit per-element values as quantized integers and finds **no meaningful compression** in streams derived directly from them; they behave almost uniformly random. Compression is therefore focused on the **scaling factors**, which are stored in **FP8** or similar higher precision and remain amenable to exponent-based compression. In the **DeepSeek-R1** example, the original dataset was **395 GB**, of which **39 GB** consisted of FP8 scalers; compressing the scalers by about half yields an overall dataset saving of **~5%**.

The abstract claim of “compression ratios up to **62% for BF16** and **83% for FP8**” is interpreted in the paper’s detailed exposition as **size reductions**, not compressed/original ratios. In that reading, **62% for BF16** corresponds to compression to **38%** of original size, and **83% for FP8** corresponds to compression to **17%** of original size for selected components.

## 7. Practical implications, misconceptions, and limitations

The two ZipNN lines address different bottlenecks but share an emphasis on preserving model utility.

For MTZ, the principal benefits are **memory/storage savings**, a potentially reduced **dynamic memory footprint**, and **low retraining overhead**. The VGG-16 experiments show that two large models can share **~40%** of parameters with **<0.5%** accuracy loss, or fully share the hidden backbone with moderate degradation; the ResNet-28 experiment shows that five models can share **~90%** of parameters with only **~0.25%** mean accuracy drop [1805.09791]. The limitations are explicit: MTZ assumes **task relatedness**, requires **architectural alignment** or at least alignable blocks, depends on the availability of good **pre-trained models**, incurs nontrivial cost for computing layer-wise Hessians, and extends to more than two networks through a **greedy sequential** procedure that is not guaranteed to be globally optimal.

For lossless-compression ZipNN, the practical applications include **model hubs**, **training checkpoints**, **distributed or federated training**, and **runtime cache management**. The 2024 paper estimates that these methods could save **over an ExaByte per month** of network traffic on a large model hub like **Hugging Face** [2411.05239]. The method is complementary to lossy compression: quantization, pruning, and distillation alter model numerics, whereas ZipNN preserves them exactly and can be applied **in addition to** quantization. The 2025 extension makes this complementarity explicit for **FP8** and **FP4**.

Several misconceptions are addressed directly by the literature. One is that generic compressors such as **gzip**, **zlib**, **Zstd**, **LZ4**, or **Snappy** should suffice; the ZipNN papers argue that neural-network tensors have little useful multi-byte repetition, but strong **field-specific skew**, especially in exponents, so structure-aware separation of fields is more effective. Another is that all low-precision formats are equally compressible; the 2025 study shows that **FP8 exponents are still skewed and compressible**, whereas raw **FP4 per-element values are essentially incompressible**, with practical gains concentrated in the scaling factors. A third is that “ZipNN” names a single technique; the arXiv record instead supports two technically separate meanings.

Taken together, the literature presents ZipNN as a pair of specialized compression paradigms. One compresses **sets of models** by learning where representations can be shared with minimal error increase; the other compresses **tensor encodings** by isolating low-entropy fields in floating-point data. The common theme is not a shared algorithmic core, but the use of neural-network-specific structure to reduce storage, transmission, or deployment cost without discarding the functions that matter.

Source: https://www.emergentmind.com/topics/zipnn