ZipNN: Model Merging & Lossless Compression

• ZipNN is an overloaded term defining both multi-task zipping, which merges pre-trained networks via layer-wise neuron sharing, and lossless compression methods that exploit floating-point bit patterns.
• The multi-task zipping framework reduces memory usage by sharing neurons across correlated models with minimal accuracy loss and significantly lower retraining overhead.
• The lossless compression approach leverages the statistical skew in floating-point exponents to achieve notable size reductions and faster compression/decompression compared to standard methods.

ZipNN is an overloaded term in the arXiv literature. In one line of work, it denotes Multi-Task Zipping (MTZ), a framework for cross-model compression that merges correlated, pre-trained neural networks by layer-wise neuron sharing and light retraining (He et al., 2018). In a later and distinct line of work, it denotes a family of lossless compression methods for neural network tensors that exploit the statistical structure of floating-point representations—especially the skew of exponent fields—to reduce storage and transmission costs while preserving exact bitwise reconstruction (Hershcovitch et al., 2024). Subsequent work extends this latter ZipNN formulation to BF16 delta checkpoints, FP8 and FP4 formats, and K/V caches used in LLMs (Heilper et al., 20 Aug 2025).

1. Dual usage of the term

The two established uses of “ZipNN” concern different objects and different compression levels.

Usage	Object being zipped	Core mechanism
Multi-Task Zipping	Multiple pre-trained networks	Layer-wise neuron sharing
ZipNN lossless compression	Floating-point tensors and model artifacts	Exponent extraction, byte grouping, entropy coding

In the MTZ sense, “zipping” means constructing a joint model $M^C$ from separately trained models $M^A, M^B, \dots$, with a shared subspace and task-specific subspaces. In the lossless-compression sense, “zipping” means representing model files, checkpoints, gradients, optimizer states, or K/V caches in a compact encoded form and then reconstructing the exact original bytes during decompression.

This suggests that the shared label is conceptual rather than methodological: one research line compresses across models by merging representations, whereas the other compresses within stored tensor representations by exploiting floating-point structure.

2. Multi-Task Zipping as cross-model compression

MTZ addresses a setting in which multiple correlated tasks are already served by separate deep neural networks, but storing and running all of them independently is prohibitive on mobile or edge devices (He et al., 2018). Traditional pruning, quantization, and low-rank factorization reduce redundancy within a single network; MTZ instead targets redundancy across networks.

The central mechanism is layer-wise neuron sharing. For corresponding hidden layers $l$ in models $M^A$ and $M^B$, MTZ identifies neurons that perform similar functions, merges selected pairs into shared neurons, and retains the remainder as task-specific neurons. If the original layers contain $N^A_l$ and $N^B_l$ neurons, then after zipping layer $l$ the combined model contains $\tilde N_l$ shared neurons, $\hat N^A_l = N^A_l - \tilde N_l$ task-$M^A, M^B, \dots$0-specific neurons, and $M^A, M^B, \dots$1 task-$M^A, M^B, \dots$2-specific neurons.

The resulting architecture is a multi-head, partially shared backbone. During inference for task $M^A, M^B, \dots$3, the shared neurons and the $M^A, M^B, \dots$4-specific neurons are active, while $M^A, M^B, \dots$5-specific connections are disabled; task $M^A, M^B, \dots$6 is handled symmetrically. The procedure is therefore not joint training from scratch with a manually chosen sharing pattern, but a post hoc transformation of pre-trained models into a single composite model.

A key practical motivation is that MTZ is designed to inherit information from each original model and then require only light retraining to re-boost task accuracy. In the VGG-16 experiments, the paper reports that MTZ can fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with <0.5% increase in the test errors for both tasks. The number of iterations required to retrain the combined network is reported as at least 17.8 times lower than that of training a single VGG-16 network.

3. MTZ mathematical formulation and zipping procedure

MTZ formalizes neuron sharing through a layer-wise error criterion. For layer $M^A, M^B, \dots$7, it defines task-specific errors before and after zipping and combines them as

$M^A, M^B, \dots$8

The change in error is approximated by a second-order Taylor expansion,

$M^A, M^B, \dots$9

with the gradient term neglected at a local minimum. Following Dong et al. (L-OBS), the method uses layer-wise Hessians rather than a full-network Hessian (He et al., 2018).

For a candidate merge between neuron $l$0 in model $l$1 and neuron $l$2 in model $l$3, only the incoming weights from already shared neurons in layer $l$4 are updated. The layer-wise Hessians reduce to covariance matrices of layer inputs, scaled by $l$5 and $l$6. Enforcing equality of the post-update incoming weight vectors yields a constrained optimization problem whose solution defines both the optimal weight adjustments and the minimum induced error increase.

This leads directly to the MTZ neuron functional difference metric

$l$7

where smaller $l$8 indicates that two neurons can be merged with smaller approximate increase in layer-wise error. The update rule

$l$9

produces the shared incoming weight vector for the merged neuron.

The algorithm operates layer by layer. Its inputs are the pre-trained weight matrices, training data for both tasks, the balance coefficient $M^A$0, and either a prescribed number of shared neurons $M^A$1 or a threshold $M^A$2 on acceptable neuron-pair error. At each hidden layer, MTZ performs a forward pass to obtain layer inputs, computes layer-wise Hessians, evaluates all candidate pairwise functional differences, selects the pairs with smallest $M^A$3 or those satisfying $M^A$4, updates the selected weights, permutes the corresponding rows in the next layer to preserve connectivity, and then performs a small number of SGD iterations to restore accuracy.

The paper interprets this as a loss-aware matching and merging procedure rather than weight averaging or heuristic alignment. The theoretical motivation is that minimizing local layer-wise error should keep the overall network deviation small, and the empirical results support that interpretation.

4. Empirical behavior of network zipping

The MTZ evaluation spans same-task model merging, cross-task model merging, and sequential zipping of multiple residual networks (He et al., 2018).

For LeNet-300-100 and LeNet-5 on MNIST, the goal was to fully share all hidden layers between differently trained models for the same task. After full hidden-layer zipping and light retraining, the reported errors were: 1.64% for dense LeNet-300-100 after starting from 1.57% and 1.60%, 1.83% for sparse LeNet-300-100 after starting from 1.80% and 1.81%, 0.93% for dense LeNet-5 after starting from 0.89% and 0.95%, and 1.29% for sparse LeNet-5 after starting from 1.27% and 1.28%. Retraining required 550, 800, 600, and 1200 iterations respectively. The paper further reports that even without retraining, fully merging both hidden layers of LeNet-300-100 incurs only ~1.5% absolute increase in test error, whereas random neuron sharing yields ~33.47% error.

For two VGG-16 networks on different tasks, task $M^A$5 was ImageNet ILSVRC-2012 classification with top-5 error ≈10.31%, and task $M^A$6 was CelebA facial attribute classification with final error ≈8.50%. In the fully shared setting, all 13 CONV layers + fc6 + fc7 were shared and only the final classification layers remained task-specific; the final errors were 14.07% top-5 on ImageNet and 11.09% on CelebA. In the adaptive sharing setting constrained to <0.5% error increase for both tasks, the final errors were 10.75% vs 10.31% baseline on ImageNet and 8.95% vs 8.50% baseline on CelebA. The overall parameter sharing was 39.61%, with 56.94% shared in convolutional layers and 38.17% shared in fc6 & fc7. Total zipping and retraining required 20,650 iterations, reported as ≥ 17.9× fewer than training one VGG-16 from scratch.

The paper also compares MTZ with a fully shared multi-task VGG-16 trained from scratch with the same topology and parameter count as the fully shared MTZ-zipped model. The from-scratch model reached 14.88% on ImageNet and 13.29% on CelebA, which was worse than MTZ under the same sharing topology. This is used to argue that zipping pre-trained models can outperform naïve joint training with a fixed sharing pattern.

For five ResNet-28 networks widened as in Zagoruyko and Komodakis, trained separately on CIFAR-100, GTSR, Omniglot, SVHN, and UCF101, MTZ sequentially zipped them with a target of 90% compression among the five. The baseline of five separate models had mean error ≈17.95% across tasks; the joint zipped model, with ≈1.5× the parameters of a single ResNet, had mean error ≈18.20%, corresponding to an average accuracy drop ≈ 0.25% absolute.

These experiments delimit MTZ’s operating regime. The method is most effective when tasks are sufficiently correlated that many neuron pairs exhibit small $M^A$7; when tasks are less related, fewer pairs satisfy the error criterion and the amount of safe sharing decreases.

5. ZipNN as lossless compression for AI models

In the later ZipNN line, the problem is no longer cross-model merging but lossless compression of model representations for storage and transmission (Hershcovitch et al., 2024). The method targets dense tensors such as weights, biases, normalization parameters, embeddings, gradients, and optimizer states in common floating-point types including BF16, FP16, and FP32. Decompression restores the exact original tensor bytes, so the logical model and datatype remain unchanged.

The core observation is that floating-point tensors are not statistically homogeneous at the bit level. A value is represented as

$M^A$8

for FP32, with analogous layouts for BF16 and FP16. In trained neural networks, exponent values occupy a narrow range: for BF16/FP32 weights in LLMs and ResNet, out of 256 possible exponent byte values, only ≈40 appear at all, and the top ≈12 exponent values account for ≈99.9% of all parameters. ZipNN therefore separates the exponent bytes from the remaining sign and mantissa bits.

The 2024 formulation organizes compression into four stages: preprocessing, bit-level reorganization, per-group compression, and reconstruction. Exponent extraction creates dedicated exponent streams; for “clean” models, ZipNN also applies byte grouping so that each byte position of the mantissa and sign is compressed separately. Compression is then performed per chunk and per byte group. The method favors Huffman-only entropy coding on exponent streams because LZ-style repetition removal contributes little on these distributions and can reduce effectiveness by adding back-pointer metadata. For zero-heavy streams, especially in deltas and some training artifacts, ZipNN can switch to Zstd. Chunk-level independence enables parallel compression and decompression and is recorded through a metadata map.

The paper distinguishes regular models from clean models. Regular models are typical trained models without heavy rounding or postprocessing; clean models have significant rounding or format conversion after training, leading to many zero or structured mantissa bits. This distinction materially affects compressibility. For regular BF16 models such as Falcon-7B, Bloom, OpenLLaMA-3B, Mistral-7B, and Llama-3.1-8B-Instruct, the reported compressed sizes cluster at 66.3%–67.4% of original size. For regular FP32 models such as wav2vec, BERT-base, and OLMo-1B, the reported compressed sizes cluster around 83.0%–83.3%. For clean FP32 models, the reported compressed sizes are substantially lower: 41.8% for XLM-RoBERTa, 48.1% for CLIP, and 33.7% for T5-base.

The comparison with general-purpose compression is central. On a 1 GB slice of Llama-3.1-8B-Instruct (BF16), Zstd produced 77.7% compressed size with compression and decompression speeds of 0.71 GB/s and 1.02 GB/s, whereas ZipNN produced 66.4% compressed size with 1.15 GB/s compression and 1.65 GB/s decompression. On OLMo-1B (FP32), ZipNN reduced size from 92.3% under Zstd to 83.2%, while increasing compression speed from 0.97 GB/s to 1.64 GB/s and decompression speed from 1.02 GB/s to 2.48 GB/s. On XLM-RoBERTa (FP32, clean), ZipNN reduced size from 57.4% under Zstd to 42.9%, with compression speed increasing from 0.18 GB/s to 0.83 GB/s and decompression speed from 0.77 GB/s to 1.41 GB/s.

The paper’s broader claim is that ZipNN “often sav[es] 33% and at times reduc[es] over 50% of the model size,” and that on popular models such as Llama 3, it achieves space savings over 17% better than vanilla compression while also improving compression and decompression speeds by 62%.

6. Low-precision extensions: BF16 deltas, FP8, FP4, and K/V caches

The 2025 extension generalizes the lossless-compression ZipNN framework to lower-precision floating-point formats and runtime artifacts (Heilper et al., 20 Aug 2025). The same structural principle is retained: decompose each tensor into sign, exponent, and mantissa streams, then apply entropy coding—especially Huffman coding—to the exponent stream, whose empirical distribution remains skewed even in lower precision.

For BF16 delta checkpoints, the pipeline first computes

$M^A$9

between consecutive checkpoints, where XOR is applied bitwise over 16-bit BF16 words. This increases the number of zeros when training is stable or adjacent checkpoints are similar. The resulting sign, exponent, and mantissa streams are then compressed per checkpoint, per layer file, and in fixed-size chunks. On llm360/Amber (6.74B parameters), the paper reports exponent-stream compression ratios as low as 0.07, mantissa ratios around 0.69–0.92, and total compressed delta checkpoint size down to 38% of the original in later training stages.

For FP8 weights, especially E4M3, each value comprises 1 sign bit, 4 exponent bits, and 3 mantissa bits. Despite the smaller exponent field, the exponent distribution remains compressible. The paper reports exponent-stream ratios in the 0.20–0.30 range, mantissa ratios around 0.80+, and overall FP8 tensor ratios of 55–70% of original size.

For K/V caches in BF16 and FP8, the paper recommends mostly static Huffman codes derived from representative cache samples, updating them only if compression degrades. Compression is performed per layer and often per head. The reported exponent compression ratios are ~0.25–0.45 for FP8 caches and <0.20 for BF16 caches, with overall K/V cache memory reductions of 20–30% and negligible impact on latency when static Huffman codes are used. The paper states that this is, to its knowledge, the first demonstration of effective lossless compression of K/V caches.

For FP4 microscaling formats, the outcome is different. The paper treats the 4-bit per-element values as quantized integers and finds no meaningful compression in streams derived directly from them; they behave almost uniformly random. Compression is therefore focused on the scaling factors, which are stored in FP8 or similar higher precision and remain amenable to exponent-based compression. In the DeepSeek-R1 example, the original dataset was 395 GB, of which 39 GB consisted of FP8 scalers; compressing the scalers by about half yields an overall dataset saving of ~5%.

The abstract claim of “compression ratios up to 62% for BF16 and 83% for FP8” is interpreted in the paper’s detailed exposition as size reductions, not compressed/original ratios. In that reading, 62% for BF16 corresponds to compression to 38% of original size, and 83% for FP8 corresponds to compression to 17% of original size for selected components.

7. Practical implications, misconceptions, and limitations

The two ZipNN lines address different bottlenecks but share an emphasis on preserving model utility.

For MTZ, the principal benefits are memory/storage savings, a potentially reduced dynamic memory footprint, and low retraining overhead. The VGG-16 experiments show that two large models can share ~40% of parameters with <0.5% accuracy loss, or fully share the hidden backbone with moderate degradation; the ResNet-28 experiment shows that five models can share ~90% of parameters with only ~0.25% mean accuracy drop (He et al., 2018). The limitations are explicit: MTZ assumes task relatedness, requires architectural alignment or at least alignable blocks, depends on the availability of good pre-trained models, incurs nontrivial cost for computing layer-wise Hessians, and extends to more than two networks through a greedy sequential procedure that is not guaranteed to be globally optimal.

For lossless-compression ZipNN, the practical applications include model hubs, training checkpoints, distributed or federated training, and runtime cache management. The 2024 paper estimates that these methods could save over an ExaByte per month of network traffic on a large model hub like Hugging Face (Hershcovitch et al., 2024). The method is complementary to lossy compression: quantization, pruning, and distillation alter model numerics, whereas ZipNN preserves them exactly and can be applied in addition to quantization. The 2025 extension makes this complementarity explicit for FP8 and FP4.

Several misconceptions are addressed directly by the literature. One is that generic compressors such as gzip, zlib, Zstd, LZ4, or Snappy should suffice; the ZipNN papers argue that neural-network tensors have little useful multi-byte repetition, but strong field-specific skew, especially in exponents, so structure-aware separation of fields is more effective. Another is that all low-precision formats are equally compressible; the 2025 study shows that FP8 exponents are still skewed and compressible, whereas raw FP4 per-element values are essentially incompressible, with practical gains concentrated in the scaling factors. A third is that “ZipNN” names a single technique; the arXiv record instead supports two technically separate meanings.

Taken together, the literature presents ZipNN as a pair of specialized compression paradigms. One compresses sets of models by learning where representations can be shared with minimal error increase; the other compresses tensor encodings by isolating low-entropy fields in floating-point data. The common theme is not a shared algorithmic core, but the use of neural-network-specific structure to reduce storage, transmission, or deployment cost without discarding the functions that matter.