---
title: 'DOTResize: Optimal Model Compression'
url: https://www.emergentmind.com/topics/dotresize
type: topic
---

# DOTResize: Optimal Model Compression

DOTResize encompasses two distinct yet complementary lines of model efficiency research: (1) width reduction in Transformers via neuron merging based on discrete optimal transport (DOT) [2507.04517], and (2) dynamic input-resolution selection for convolutional neural networks (CNNs) using per-instance compute-accuracy optimization, as grounded in the Dynamic Resolution Network (DRNet) methodology [2106.02898]. Both approaches use the term "DOTResize" to denote data- and structure-aware reductions of model redundancy, targeting either internal representation (width) or external input adaptation (resolution), each underpinned by mathematically principled frameworks.

## 1. Neuron Merging in Transformers via Discrete Optimal Transport

DOTResize for Transformer compression reframes width reduction as a discrete optimal transport problem, targeting neuron-level redundancies and merging similar neurons without discarding information. Given a neural layer with hidden-width \(d_{\rm orig}\), the goal is to reduce the width to \(d_{\rm new} < d_{\rm orig}\) by merging the entire neuron inventory onto a smaller basis. The set of original neurons is treated as a discrete 'source' distribution, and a chosen subset (e.g., neurons with highest mean \(\ell_2\)-activations) forms the 'target' distribution. The method utilizes calibration data to gather activations \(X \in \mathbb{R}^{t \times d_{\rm orig}}\) and constructs a ground-cost matrix \(C_{i,j} = \| x_i - (X_S)_j \|_1\), where \(S\) is the target neuron index set.

The unregularized discrete OT problem seeks a transport plan \(T \in \mathbb{R}_+^{d_{\rm orig} \times d_{\rm new}}\), minimizing \(\langle T, C \rangle\) under marginal constraints defined by the source and target distributions. Empirically, entropic regularization (Sinkhorn) is introduced,
\[
T^* = \arg\min_{T \geq 0, T1=b, T^T1=a} \langle T, C \rangle - \lambda H(T)
\]
where \(H(T)\) denotes entropy and \(\lambda\) regulates 'softness' versus sparsity. Practical integration with linear weight matrices requires rescaling of \(T\) so its rows sum to one.

## 2. Entropic OT and Integration with Transformer Normalization

The use of Sinkhorn-Knopp iterations allows efficient approximation of the entropic OT solution. For each Transformer layer, the OT map is factored via thin QR decomposition:
\[
M = Q R, \qquad Q^T Q = I
\]
Defining \(M_{\text{proj}} = Q\) and \(M_{\text{unproj}} = R M^\dagger\) (with \(M^\dagger\) as the (pseudo-)inverse), this decomposition ensures invariance of pre-norm RMSNorm layers:
\[
\mathrm{RMSNorm}(x Q) R M^\dagger = \mathrm{RMSNorm}(x)
\]
Thus, the insertion of \(M_{\text{proj}}\) and \(M_{\text{unproj}}\) into attention and feed-forward projections does not disrupt pre-norm stability, allowing for lossless transformation of signal through reduced-width layers.

## 3. Workflow, Algorithmic Complexity, and Implementation

The DOTResize process is a one-shot, training-free pipeline. For each layer:

1. Collect activations for calibration data.
2. Select top \(d_{\rm new}\) neurons (mean \(\ell_2\)-norm).
3. Compute OT cost and transport plan via Sinkhorn iterations.
4. Apply QR decomposition and generate project/unproject maps.
5. Absorb the factors into all linear sub-layers.
6. Output compressed weights with new width.

The total computational overhead is dominated by the OT (Sinkhorn) and QR decomposition, scaling as \(O(K d_{\rm orig} d_{\rm new})\) and \(O(d_{\rm new}^2 d_{\rm orig})\), respectively. Empirical use of \(t \approx 2^{18}\) tokens and \(K \approx 200\) iterations suffices for rapid compression on a single GPU per layer. On state-of-the-art LLMs (e.g., Llama-3.1 70B), inference device requirements drop from 8 to 6 V100 GPUs for 30% width reduction.

## 4. DOTResize versus Structured Pruning Strategies

Whereas structured pruning discards 'least important' neurons, rerunning inference on a struct subnetwork, DOTResize merges the entire activation basis onto a smaller set, redistributing all useful information. This approach avoids the sharp performance drop observed in classical pruning and achieves superior utilization of pruned-out capacity by explicitly reprojecting lost signals. Quantitatively, DOTResize consistently recovers 2–5 perplexity points over pruning baselines and outperforms state-of-the-art SliceGPT, especially when composed with a PCA pre-projection.

## 5. Empirical Evaluation and Benchmarks

Empirical language modeling and zero-shot task experiments demonstrate robust performance of DOTResize. On Wikitext-2 (20–30% reduction):

| Model             | Magnitude-prune | DOTResize | SliceGPT | PCA+DOTResize |
|-------------------|----------------|-----------|----------|---------------|
| Llama-3.1-8B      | 29.33 / 108.23 | 16.57 / 36.20 | 11.25 / 19.42 | 10.24 / 16.46 |

On zero-shot tasks at 20% sparsity, for Mistral-12B and Phi-4-12B, PCA+DOTResize provides +1–12 percentage points accuracy relative to SliceGPT. The method is effective across models and is easily composable with quantization schemes such as 4-bit post-training quantization for further size and speed improvements.

## 6. Dynamic-Resolution DOTResize for CNNs

The DRNet-based DOTResize module addresses compute redundancy in CNNs by dynamically selecting the smallest input resolution per image that maintains (or improves) accuracy. A lightweight resolution predictor is trained jointly with the backbone, using a candidate resolution set \(R = \{ r_1, ..., r_m \}\) (e.g., \(R=[224,168,112]\)). The predictor's output, after softmax and straight-through Gumbel–Softmax reparameterization, yields a one-hot allocation to the resolution for each image. Each candidate resolution is handled by a private batch normalization branch. The loss combines cross-entropy and a compute regularizer that constrains average FLOPs to a target \(\alpha\).

Practical integration requires minimal code overhead, leveraging on-the-fly resizing, BN multiplexing, and backpropagation through multitask losses. The system is compatible with any off-the-shelf CNN backbone.

Empirically, DRNet-based DOTResize achieves notable compute reductions without accuracy loss:

| Model                    | Params | FLOPs | \(\downarrow\)FLOPs | Top-1 | Top-5 |
|--------------------------|--------|-------|------------------|-------|-------|
| ResNet-50 (224)          | 25.6M  | 4.1G  | —                | 76.1% | 92.9% |
| DOTResize (default)      | 30.5M  | 3.7G  | 10%              | 77.5% | 93.5% |
| DOTResize (\(\alpha\)=2.5)| 30.5M | 2.7G  | 34%              | 76.2% | 92.8% |

On MobileNetV2, DOTResize similarly outperforms fixed lower-resolution models.

## 7. Hyperparameters, Practical Tips, and Extensions

For Transformer DOTResize, Sinkhorn regularization \(\lambda \in [0.05, 0.5]\) (default 0.1) is effective, and 2¹⁷–2¹⁹ calibration tokens are sufficient. For CNN DOTResize, three-level resolution search-spaces provide most of the gain, and hyperparameter choices (\(\alpha\), \(\eta\), annealed \(\tau\)) allow fine control of the FLOPs-accuracy tradeoff. OT and QR can be parallelized per-layer and per-GPU. Minimal engineering is required: 20–50 lines of predictor code, built-in PyTorch resizing, and batchnorm multiplexing.

A plausible implication is that DOTResize establishes a general paradigm for resource-efficient inference, where discrete optimal transport underlies principled width reduction and dynamic instance-aware input adaptation for deep neural networks. Both methodologies provide compute savings (10–44%) and, in tested cases, maintain or improve accuracy relative to baseline and state-of-the-art compression/pruning methods [2507.04517, 2106.02898].

Source: https://www.emergentmind.com/topics/dotresize