---
title: Locality-Aware Redundancy Pruning (LoRP)
url: https://www.emergentmind.com/topics/locality-aware-redundancy-pruning-lorp
type: topic
---

# Locality-Aware Redundancy Pruning (LoRP)

Searching arXiv for the cited LoRP and adjacent pruning papers to ground the article.
Locality-Aware Redundancy Pruning (LoRP) is a **training-free, one-shot depth pruning framework** for large language models that uses global inter-layer hidden-state similarity to decide how pruning should be distributed across network depth [2605.27786]. In the LoRP formulation, “locality-aware” refers to awareness of **how localized or distributed representational redundancy is across depth** for a given architecture, while “redundancy pruning” refers to removing layers whose representations overlap strongly with others inside representational clusters [2605.27786]. The method is motivated by the claim that **inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture**, so pruning should not rely only on isolated layer scores or on a fixed belief that redundancy is contiguous [2605.27786]. Related work uses the same broad design intuition in other domains—view-space Gaussian pruning in 3DGS-SLAM [2606.24796], shallow-layer token redundancy pruning in Vision Transformers [2606.08156], and spatially modulated token redundancy pruning in VLMs [2512.02700]—but LoRP itself is explicitly formulated for decoder-only LLM depth compression [2605.27786].

## 1. Conceptual basis and problem setting

LoRP is introduced for pretrained decoder-only Transformers viewed as deep compositions of Transformer blocks,
\[
f_{\theta} = f_N \circ f_{N-1} \circ \cdots \circ f_1,
\]
where \(N\) is the number of Transformer blocks and \(f_l\) is the \(l\)-th block [2605.27786]. Given a pruning budget \(k \ll N\), depth pruning seeks a removable layer set
\[
\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,
\]
such that the pruned model preserves the original model’s function as much as possible while reducing compute and memory [2605.27786]. The ideal but intractable objective is
\[
\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \\ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),
\]
where \(\mathcal{L}(\cdot,\cdot)\) measures functional discrepancy between the dense and pruned networks [2605.27786].

The method’s core empirical premise is **representational redundancy across network depth**: multiple layers may produce highly similar hidden representations, so one of them may be removable with limited harm [2605.27786]. The paper’s main critique of prior one-shot pruning is twofold. First, **local layer-importance methods** score layers independently. Second, **contiguous-region methods** assume redundancy is concentrated in adjacent depth regions. LoRP argues that both assumptions can fail because the organization of redundancy is architecture-dependent [2605.27786].

The paper reports that some model families exhibit **localized redundancy**—similarity concentrated among nearby layers—while others exhibit **globally distributed redundancy**—similarity spread more broadly across the network, including non-adjacent layers [2605.27786]. This motivates a pruning strategy that first measures the geometry of inter-layer similarity and then allocates pruning accordingly. In this sense, LoRP is not merely a ranking rule for individual layers; it is a framework for deciding whether pruning should concentrate in a few regions or spread across several representational groups [2605.27786].

A plausible implication is that LoRP’s “locality” is neither token-local nor spatial in the vision sense. It is locality **along depth**, defined by the extent to which redundancy decays with layer distance. This distinguishes LoRP from methods that use explicit image neighborhoods or graph neighborhoods, even when they share the broader intuition that pruning should follow local redundancy structure rather than global heuristics.

## 2. Representation Locality Score and similarity estimation

LoRP characterizes architecture-dependent redundancy structure through the **Representation Locality Score (RLS)** [2605.27786]. The method first collects the **input hidden states to Transformer blocks**, using a small unlabeled calibration corpus
\[
\mathcal{D} = \{x_m\}_{m=1}^M.
\]
For token position \(t\) in sample \(x_m\),
\[
\mathbf{h}_{l,m,t} \in \mathbb{R}^{d}
\]
denotes the hidden representation entering layer \(l\) [2605.27786].

Each token representation is \(\ell_2\)-normalized:
\[
\hat{\mathbf{h}}_{l,m,t} = \frac{\mathbf{h}_{l,m,t}}{\|\mathbf{h}_{l,m,t}\|_2 + \epsilon},
\]
and pairwise layer similarity is computed as the mean per-token cosine similarity between layer-input hidden states:
\[
S_{ij} = \frac{1}{\sum_{m=1}^{M} T_m} \sum_{m=1}^{M} \sum_{t=1}^{T_m} \hat{\mathbf{h}}_{i,m,t}^{\top} \hat{\mathbf{h}}_{j,m,t},
\]
where \(T_m\) is the sequence length of sample \(x_m\) [2605.27786]. The resulting matrix \(\mathbf{S}\in\mathbb{R}^{N\times N}\) is the global inter-layer similarity matrix.

From this, LoRP defines the global off-diagonal mean similarity:
\[
\bar{S}_{\mathrm{off}} = \frac{2}{N(N-1)} \sum_{i<j} S_{ij},
\]
and then the **Representation Locality Score**:
\[
\mathrm{RLS}(\mathbf{S}) = -\log_2 \bar{S}_{\mathrm{off}}.
\]
The paper emphasizes that RLS uses the **global off-diagonal mean rather than only near-diagonal similarity**, to avoid bias toward assuming local redundancy [2605.27786].

The interpretation is explicit. **High RLS** implies lower global off-diagonal similarity, faster decay with depth distance, and therefore more **localized** redundancy. **Low RLS** implies stronger similarity among distant layers and therefore more **globally distributed** redundancy [2605.27786]. Reported RLS values are: Llama-3.1-8B \(1.149\), OLMo-3-7B \(0.941\), Mistral-Nemo-12B \(0.926\), Qwen3-8B \(0.685\), and Qwen3-14B \(0.644\) [2605.27786]. These values support the paper’s claim that Qwen is more globally redundant, while Llama is more localized.

The calibration setup used experimentally is fixed: **128 sequences**, **sequence length 2048**, sampled from **C4** [2605.27786]. This is sufficient for LoRP’s one-shot similarity estimation, and no recovery training is used afterward. The paper intentionally studies LoRP in a **no-recovery** setting [2605.27786].

## 3. Clustering, redundancy modeling, and pruning allocation

After computing RLS, LoRP converts similarity to nonnegative affinity for clustering:
\[
\mathbf{A} = \frac{\mathbf{S}+1}{2}.
\]
It then applies **spectral clustering with precomputed affinity** to partition layers into representational clusters
\[
\mathcal{C} = \{C_1, C_2, \dots, C_K\}
\]
[2605.27786]. The clusters are re-indexed by first-occurrence depth. The paper explicitly states that clusters are **not necessarily contiguous**, which is a key design choice because spectral clustering can reflect **non-adjacent representational similarity without assuming layer contiguity** [2605.27786].

The number of clusters \(K\) is chosen by an RLS-guided heuristic:
- \(\mathrm{RLS} \ge 1.0 \Rightarrow K=2\)
- \(0.7 \le \mathrm{RLS} < 1.0 \Rightarrow K=3\)
- \(\mathrm{RLS} < 0.7 \Rightarrow K=4\)

Thus lower RLS leads to finer clustering and encourages pruning spread across more groups [2605.27786]. The reported cluster counts used experimentally are \(K=2\) for LLaMA-3.1-8B, \(K=3\) for OLMo-3-7B and Mistral-Nemo-12B, and \(K=4\) for Qwen3-8B and Qwen3-14B [2605.27786].

Within each cluster \(C_k\), LoRP defines a **layer-level redundancy score**:
\[
r(l) = \frac{1}{|C_k|-1} \sum_{\substack{m\in C_k \\ m\neq l}} S_{lm}.
\]
This measures how redundant layer \(l\) is relative to other members of its cluster [2605.27786]. Higher \(r(l)\) means stronger overlap and therefore higher removability. Boundary layers
\[
\mathcal{B} = \{1, N\}
\]
are excluded from the pruning candidate set [2605.27786].

LoRP then allocates pruning in two stages. In **Stage 1 — coverage-aware initialization**, the method ensures that every representational region is considered. For each cluster,
\[
l_k^* = \arg\max_{l\in C_k \setminus \mathcal{B}} r(l),
\]
and the initial pruning pool is
\[
\mathcal{P}_0 = \bigcup_{k=1}^{K}\{l_k^*\}.
\]
If the total pruning budget is smaller than the number of clusters, only the top-ranked candidates are kept according to intra-cluster redundancy [2605.27786]. The paper states that this stage prevents “pruning collapse,” where all removals happen in one region simply because it initially appears most redundant.

In **Stage 2 — residual redundancy allocation**, LoRP repeatedly assigns the remaining pruning budget to the cluster with greatest residual redundancy. For each cluster,
\[
\widetilde{C}_k = C_k \setminus \mathcal{P} \setminus \mathcal{B},
\]
and residual cluster redundancy is
\[
\mu_k = \frac{2}{|\widetilde{C}_k|(|\widetilde{C}_k|-1)} \sum_{\substack{i,j\in \widetilde{C}_k \\ i<j}} S_{ij}.
\]
If \(|\widetilde{C}_k|<2\), the paper sets
\[
\mu_k = -\infty.
\]
At each iteration, LoRP chooses
\[
k^* = \arg\max_k \mu_k,
\]
then removes the most redundant remaining layer from that cluster:
\[
l^* = \arg\max_{l\in \widetilde{C}_{k^*}} r(l),
\]
updates
\[
\mathcal{P} \leftarrow \mathcal{P} \cup \{l^*\},
\]
and repeats until \(|\mathcal{P}| = k\) [2605.27786].

This two-stage allocation is central to the meaning of LoRP. It does not fix local budgets beforehand. Instead, it uses global representational geometry to decide whether pruning should remain concentrated in one cluster or spread across multiple clusters. This suggests that LoRP’s locality-awareness is better understood as **cluster-aware residual redundancy allocation** rather than a purely local saliency heuristic.

## 4. Pruning action, efficiency, and empirical results

The final pruning action is direct: layers are **removed directly** by deleting the corresponding Transformer blocks from the model’s layer list and updating the configured layer count [2605.27786]. There is no merging, replacement, or auxiliary recovery module. Because the model is a sequential residual stack, deleting a block means the hidden state simply flows from the previous retained block to the next retained block [2605.27786].

The paper evaluates five open-source LLMs:
- **LLaMA-3.1-8B** (32 layers)
- **OLMo-3-7B** (32 layers)
- **Mistral-Nemo-12B** (40 layers)
- **Qwen3-8B** (36 layers)
- **Qwen3-14B** (40 layers)

Perplexity is evaluated on WikiText-2, C4, and Penn Treebank with sequence length 2048, and downstream zero-shot reasoning on nine tasks using lm-evaluation-harness [2605.27786]. The training-free depth-pruning baselines are **ShortGPT**, **LLM-Streamline**, and **LaCo** [2605.27786].

Representative perplexity results show that LoRP is usually the strongest training-free method after pruning. On LLaMA-3.1-8B at \(7/32\), average perplexity is \(41.95\) for LoRP, versus \(67.21\) for ShortGPT, \(77.56\) for LaCo, and \(\ge 2000\) for LLM-Streamline [2605.27786]. On Qwen3-8B at \(7/36\), LoRP gives \(27.92\), while LLM-Streamline gives \(229.11\), LaCo \(189.61\), and ShortGPT \(262.67\) [2605.27786]. On Qwen3-14B at \(11/40\), LoRP gives \(37.44\), versus \(276.50\) for ShortGPT, \(1024.68\) for LaCo, and \(\ge 2000\) for LLM-Streamline [2605.27786].

Downstream task accuracy is also generally better preserved. On Qwen3-8B, dense accuracy is \(68.06\); at \(7/36\), LoRP gives \(56.65\), compared with \(55.82\) for LaCo, \(54.07\) for LLM-Streamline, and \(51.50\) for ShortGPT [2605.27786]. On Qwen3-14B, dense accuracy is \(71.22\); at \(11/40\), LoRP gives \(52.84\), compared with \(50.53\), \(47.59\), and \(46.51\) for the three baselines [2605.27786].

The appendix reports practical efficiency on **Qwen3-14B**. Dense latency is \(676.7 \pm 2.5\) ms. LoRP gives \(497.4 \pm 1.5\) ms at \(L_p=11\) (**1.36×**), \(465.3 \pm 1.4\) ms at \(L_p=13\) (**1.45×**), and \(432.8 \pm 1.1\) ms at \(L_p=15\) (**1.56×**) [2605.27786]. Peak GPU memory drops from \(28.12\) GiB dense to \(21.35\) GiB (**−24.1%**), \(20.12\) GiB (**−28.4%**), and \(18.89\) GiB (**−32.8%**) at the same pruning levels [2605.27786].

The strongest empirical pattern is that LoRP’s gains are especially pronounced on **Qwen**, which the paper identifies as having **globally distributed redundancy** [2605.27786]. This directly supports the method’s claim that architecture-sensitive locality modeling matters. The paper also compares LoRP against iterative baselines such as **Shortened LLaMA** and **SLEB**, noting that iterative methods can achieve lower perplexity on the calibration-aligned objective, whereas LoRP often preserves higher downstream accuracy, especially on Qwen [2605.27786].

## 5. Relation to adjacent “LoRP-like” methods

Several recent papers instantiate related ideas under different notions of locality and redundancy. These do not define LoRP in the same sense, but they illuminate the design space.

In 3DGS-SLAM, "Pocket-SLAM: Rendering-Area-Aware Pruning for Memory-Efficient 3DGS-SLAM" [2606.24796] is described as **redundancy-aware and weakly locality-aware, but not in the strongest LoRP sense**. It prunes Gaussians by their effective rendering-area contribution,
\[
C_i=\sum_{\mathbf p\in\Omega}\alpha_{i}(\mathbf p), \qquad S_i=\frac{C_i}{\sum_j C_j},
\]
and then applies per-tile survivor budgets
\[
B_k^{\text{trk}}=\operatorname{clip}\!\left( \Big\lceil N_{\text{tar}}\cdot \tfrac{G_k}{\sum_j G_j}\Big\rceil, B_{\min}, B_{\max} \right)
\]
based on tile-level tracking gradients [2606.24796]. Its locality is **image-tile locality**, not explicit 3D neighborhood locality, and its redundancy score is unary rather than overlap-aware [2606.24796]. The paper explicitly notes that a stronger LoRP extension could define local neighborhoods in 3D or projected footprint overlap neighborhoods and estimate each Gaussian’s unique coverage rather than total coverage [2606.24796]. This suggests a distinction between **region-sensitive budgeting** and **explicit local mutual redundancy modeling**.

In Vision Transformers, "RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT" [2606.08156] contains a **Redundancy-Similarity Aware Pruning** stage that is the closest match to a LoRP-like component in that paper. For source token \(i\),
\[
\mathrm{score}_i = \sum_j \max(s_{ij} - \tau, 0), \qquad \tau = 0.7,
\]
and the method prunes the top-\(r\) source tokens with the highest redundancy scores in shallow-to-middle layers [2606.08156]. The paper explicitly states that locality is **implicit rather than explicitly geometric**: early layers encode local patterns and textures, but the algorithm uses bipartite similarity among token groups rather than spatial neighborhoods [2606.08156]. This makes RAPID **redundancy-aware and early-layer local-pattern-aware**, but not explicitly neighborhood-based in the spatial sense [2606.08156].

In VLM pruning, "VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm" [2512.02700] is much closer to an explicit locality-aware redundancy formulation. It modulates feature-space similarity using nearest-selected spatial distance:
\[
\delta_i(S) = \min_{j\in S} D^{(\mathrm{sp})}_{ij}, \qquad
\bar{\delta}_i(S) = \delta_i(S)/D_{\max},
\]
\[
\widetilde{M}_{ij} = M_{ij}\,\bigl(1+\lambda\,\bar{\delta}_i(S)\bigr),
\]
with \(\lambda = 0.5\), and defines a non-duplication score
\[
r_i = 1 - \max_{j\in\mathcal S}\widetilde M_{ij}
\]
for near-to-far greedy selection [2512.02700]. The paper explicitly frames this as balancing redundancy reduction with **spatial coverage coherence**, and reports that replacing its Stage 2 with redundancy-only baselines hurts substantially [2512.02700]. A plausible implication is that VLM-Pruner is closer to a generic “Locality-Aware Redundancy Pruning” concept than LoRP’s layer-axis formulation, but it operates on visual tokens rather than network depth.

These comparisons clarify that LoRP is one member of a broader design family. Shared motifs include moving beyond scalar per-unit heuristics, conditioning pruning on local or regional structure, and defining redundancy in terms of mutual representational overlap rather than isolated saliency. The specific notion of locality, however, changes sharply across domains: depth-axis locality in LLMs [2605.27786], image-tile locality in SLAM [2606.24796], shallow-layer local-pattern locality in ViTs [2606.08156], and 2D or 3D token-grid locality in VLMs [2512.02700].

## 6. Limitations, misconceptions, and future directions

LoRP’s limitations are stated explicitly. It is evaluated only on several **decoder-only** Transformer families, and the RLS-to-\(K\) policy is **empirically derived**, not theoretically optimal [2605.27786]. The number of clusters \(K\) is still manually specified through a heuristic thresholding scheme, the method is studied in a **no-recovery** setting, and calibration uses a fixed C4 setup, leaving stability across domains open [2605.27786]. The paper also does not provide detailed ablations on alternative similarity metrics, calibration set size sweeps, token-position pooling variants, alternative clustering algorithms, or explicit Stage-1 versus Stage-2 removal comparisons [2605.27786].

A common misconception would be to read “locality-aware” as meaning that LoRP prefers neighboring layers by construction. The paper explicitly avoids that assumption. RLS is derived from the **global off-diagonal mean** of the full similarity matrix, and spectral clustering is chosen precisely because it can group **non-adjacent representationally similar layers** [2605.27786]. Locality in LoRP therefore means **how localized redundancy happens to be**, not a hard contiguity prior.

Another possible misconception is that LoRP performs a purely local layer-importance ranking. It does not. The framework first infers a global representation geometry, then clusters layers, then uses both layer-level and cluster-level redundancy to allocate pruning [2605.27786]. The pruning objective is therefore relational and architecture-sensitive, even though the final action is simple direct block deletion.

The broader literature suggests several plausible extensions. Pocket-SLAM explicitly identifies the absence of overlap-aware conditional redundancy modeling as a limitation relative to a fuller LoRP formulation [2606.24796]. RAPID notes that a more principled criterion for switching from pruning to merging is needed [2606.08156]. VLM-Pruner shows that redundancy-only pruning can become spatially fragmented unless locality enters the scoring rule itself [2512.02700]. Taken together, these works suggest that stronger future LoRP-style methods might combine global redundancy geometry with conditional or overlap-aware redundancy estimates, adaptive locality definitions, or post-pruning recovery.

The clean conclusion is that LoRP, in its explicit 2026 formulation, is a **global, architecture-aware, training-free one-shot method for depth pruning** that replaces fixed local heuristics with a representation-geometric view of redundancy [2605.27786]. Its defining technical pieces are pairwise layer-input cosine similarity, the Representation Locality Score,
\[
\mathrm{RLS}(\mathbf{S}) = -\log_2\!\left(\frac{2}{N(N-1)}\sum_{i<j} S_{ij}\right),
\]
spectral clustering on
\[
\mathbf{A}=\frac{\mathbf{S}+1}{2},
\]
the intra-cluster layer redundancy score
\[
r(l)=\frac{1}{|C_k|-1}\sum_{m\in C_k,\; m\neq l} S_{lm},
\]
and residual cluster redundancy
\[
\mu_k
=
\frac{2}{|\widetilde{C}_k|(|\widetilde{C}_k|-1)}
\sum_{i<j,\; i,j\in\widetilde{C}_k} S_{ij}.
\]
Empirically, it is especially effective on architectures like **Qwen**, where redundancy is globally distributed and prior local or contiguous pruning assumptions break down [2605.27786].

Source: https://www.emergentmind.com/topics/locality-aware-redundancy-pruning-lorp