Locality-Aware Redundancy Pruning (LoRP)

• The paper introduces LoRP as a training-free, one-shot method leveraging global inter-layer cosine similarity to identify and remove redundant Transformer layers.
• LoRP defines a Representation Locality Score and employs spectral clustering to capture non-adjacent representational redundancy, enabling adaptive pruning across network depth.
• Empirical results show that LoRP reduces compute and memory demands while preserving performance, particularly in architectures with both localized and globally distributed redundancy.

Searching arXiv for the cited LoRP and adjacent pruning papers to ground the article. Locality-Aware Redundancy Pruning (LoRP) is a training-free, one-shot depth pruning framework for LLMs that uses global inter-layer hidden-state similarity to decide how pruning should be distributed across network depth (Yun et al., 27 May 2026). In the LoRP formulation, “locality-aware” refers to awareness of how localized or distributed representational redundancy is across depth for a given architecture, while “redundancy pruning” refers to removing layers whose representations overlap strongly with others inside representational clusters (Yun et al., 27 May 2026). The method is motivated by the claim that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture, so pruning should not rely only on isolated layer scores or on a fixed belief that redundancy is contiguous (Yun et al., 27 May 2026). Related work uses the same broad design intuition in other domains—view-space Gaussian pruning in 3DGS-SLAM (Li et al., 23 Jun 2026), shallow-layer token redundancy pruning in Vision Transformers (Choi et al., 6 Jun 2026), and spatially modulated token redundancy pruning in VLMs (Wu et al., 2 Dec 2025)—but LoRP itself is explicitly formulated for decoder-only LLM depth compression (Yun et al., 27 May 2026).

1. Conceptual basis and problem setting

LoRP is introduced for pretrained decoder-only Transformers viewed as deep compositions of Transformer blocks,

$$f_{\theta} = f_N \circ f_{N-1} \circ \cdots \circ f_1,$$

where $N$ is the number of Transformer blocks and $f_l$ is the $l$-th block (Yun et al., 27 May 2026). Given a pruning budget $k \ll N$, depth pruning seeks a removable layer set

$$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$

such that the pruned model preserves the original model’s function as much as possible while reducing compute and memory (Yun et al., 27 May 2026). The ideal but intractable objective is

$$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$

where $\mathcal{L}(\cdot,\cdot)$ measures functional discrepancy between the dense and pruned networks (Yun et al., 27 May 2026).

The method’s core empirical premise is representational redundancy across network depth: multiple layers may produce highly similar hidden representations, so one of them may be removable with limited harm (Yun et al., 27 May 2026). The paper’s main critique of prior one-shot pruning is twofold. First, local layer-importance methods score layers independently. Second, contiguous-region methods assume redundancy is concentrated in adjacent depth regions. LoRP argues that both assumptions can fail because the organization of redundancy is architecture-dependent (Yun et al., 27 May 2026).

The paper reports that some model families exhibit localized redundancy—similarity concentrated among nearby layers—while others exhibit globally distributed redundancy—similarity spread more broadly across the network, including non-adjacent layers (Yun et al., 27 May 2026). This motivates a pruning strategy that first measures the geometry of inter-layer similarity and then allocates pruning accordingly. In this sense, LoRP is not merely a ranking rule for individual layers; it is a framework for deciding whether pruning should concentrate in a few regions or spread across several representational groups (Yun et al., 27 May 2026).

A plausible implication is that LoRP’s “locality” is neither token-local nor spatial in the vision sense. It is locality along depth, defined by the extent to which redundancy decays with layer distance. This distinguishes LoRP from methods that use explicit image neighborhoods or graph neighborhoods, even when they share the broader intuition that pruning should follow local redundancy structure rather than global heuristics.

2. Representation Locality Score and similarity estimation

LoRP characterizes architecture-dependent redundancy structure through the Representation Locality Score (RLS) (Yun et al., 27 May 2026). The method first collects the input hidden states to Transformer blocks, using a small unlabeled calibration corpus

$\mathcal{D} = \{x_m\}_{m=1}^M.$

For token position $t$ in sample $N$0,

$N$1

denotes the hidden representation entering layer $N$2 (Yun et al., 27 May 2026).

Each token representation is $N$3-normalized: $N$4 and pairwise layer similarity is computed as the mean per-token cosine similarity between layer-input hidden states: $N$5 where $N$6 is the sequence length of sample $N$7 (Yun et al., 27 May 2026). The resulting matrix $N$8 is the global inter-layer similarity matrix.

From this, LoRP defines the global off-diagonal mean similarity: $N$9 and then the Representation Locality Score: $f_l$0 The paper emphasizes that RLS uses the global off-diagonal mean rather than only near-diagonal similarity, to avoid bias toward assuming local redundancy (Yun et al., 27 May 2026).

The interpretation is explicit. High RLS implies lower global off-diagonal similarity, faster decay with depth distance, and therefore more localized redundancy. Low RLS implies stronger similarity among distant layers and therefore more globally distributed redundancy (Yun et al., 27 May 2026). Reported RLS values are: Llama-3.1-8B $f_l$1, OLMo-3-7B $f_l$2, Mistral-Nemo-12B $f_l$3, Qwen3-8B $f_l$4, and Qwen3-14B $f_l$5 (Yun et al., 27 May 2026). These values support the paper’s claim that Qwen is more globally redundant, while Llama is more localized.

The calibration setup used experimentally is fixed: 128 sequences, sequence length 2048, sampled from C4 (Yun et al., 27 May 2026). This is sufficient for LoRP’s one-shot similarity estimation, and no recovery training is used afterward. The paper intentionally studies LoRP in a no-recovery setting (Yun et al., 27 May 2026).

3. Clustering, redundancy modeling, and pruning allocation

After computing RLS, LoRP converts similarity to nonnegative affinity for clustering: $f_l$6 It then applies spectral clustering with precomputed affinity to partition layers into representational clusters

$f_l$7

(Yun et al., 27 May 2026). The clusters are re-indexed by first-occurrence depth. The paper explicitly states that clusters are not necessarily contiguous, which is a key design choice because spectral clustering can reflect non-adjacent representational similarity without assuming layer contiguity (Yun et al., 27 May 2026).

The number of clusters $f_l$8 is chosen by an RLS-guided heuristic:

• $f_l$9
• $l$0
• $l$1

Thus lower RLS leads to finer clustering and encourages pruning spread across more groups (Yun et al., 27 May 2026). The reported cluster counts used experimentally are $l$2 for LLaMA-3.1-8B, $l$3 for OLMo-3-7B and Mistral-Nemo-12B, and $l$4 for Qwen3-8B and Qwen3-14B (Yun et al., 27 May 2026).

Within each cluster $l$5, LoRP defines a layer-level redundancy score: $l$6 This measures how redundant layer $l$7 is relative to other members of its cluster (Yun et al., 27 May 2026). Higher $l$8 means stronger overlap and therefore higher removability. Boundary layers

$l$9

are excluded from the pruning candidate set (Yun et al., 27 May 2026).

LoRP then allocates pruning in two stages. In Stage 1 — coverage-aware initialization, the method ensures that every representational region is considered. For each cluster,

$k \ll N$0

and the initial pruning pool is

$k \ll N$1

If the total pruning budget is smaller than the number of clusters, only the top-ranked candidates are kept according to intra-cluster redundancy (Yun et al., 27 May 2026). The paper states that this stage prevents “pruning collapse,” where all removals happen in one region simply because it initially appears most redundant.

In Stage 2 — residual redundancy allocation, LoRP repeatedly assigns the remaining pruning budget to the cluster with greatest residual redundancy. For each cluster,

$k \ll N$2

and residual cluster redundancy is

$k \ll N$3

If $k \ll N$4, the paper sets

$k \ll N$5

At each iteration, LoRP chooses

$k \ll N$6

then removes the most redundant remaining layer from that cluster: $k \ll N$7 updates

$k \ll N$8

and repeats until $k \ll N$9 (Yun et al., 27 May 2026).

This two-stage allocation is central to the meaning of LoRP. It does not fix local budgets beforehand. Instead, it uses global representational geometry to decide whether pruning should remain concentrated in one cluster or spread across multiple clusters. This suggests that LoRP’s locality-awareness is better understood as cluster-aware residual redundancy allocation rather than a purely local saliency heuristic.

4. Pruning action, efficiency, and empirical results

The final pruning action is direct: layers are removed directly by deleting the corresponding Transformer blocks from the model’s layer list and updating the configured layer count (Yun et al., 27 May 2026). There is no merging, replacement, or auxiliary recovery module. Because the model is a sequential residual stack, deleting a block means the hidden state simply flows from the previous retained block to the next retained block (Yun et al., 27 May 2026).

The paper evaluates five open-source LLMs:

• LLaMA-3.1-8B (32 layers)
• OLMo-3-7B (32 layers)
• Mistral-Nemo-12B (40 layers)
• Qwen3-8B (36 layers)
• Qwen3-14B (40 layers)

Perplexity is evaluated on WikiText-2, C4, and Penn Treebank with sequence length 2048, and downstream zero-shot reasoning on nine tasks using lm-evaluation-harness (Yun et al., 27 May 2026). The training-free depth-pruning baselines are ShortGPT, LLM-Streamline, and LaCo (Yun et al., 27 May 2026).

Representative perplexity results show that LoRP is usually the strongest training-free method after pruning. On LLaMA-3.1-8B at $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$0, average perplexity is $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$1 for LoRP, versus $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$2 for ShortGPT, $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$3 for LaCo, and $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$4 for LLM-Streamline (Yun et al., 27 May 2026). On Qwen3-8B at $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$5, LoRP gives $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$6, while LLM-Streamline gives $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$7, LaCo $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$8, and ShortGPT $$\mathcal{P} \subseteq \{1,\dots,N\}, \qquad |\mathcal{P}|=k,$$9 (Yun et al., 27 May 2026). On Qwen3-14B at $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$0, LoRP gives $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$1, versus $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$2 for ShortGPT, $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$3 for LaCo, and $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$4 for LLM-Streamline (Yun et al., 27 May 2026).

Downstream task accuracy is also generally better preserved. On Qwen3-8B, dense accuracy is $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$5; at $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$6, LoRP gives $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$7, compared with $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$8 for LaCo, $$\mathcal{P}^{*} = \arg\min_{\substack{\mathcal{P}\subseteq\{1,\dots,N\} \ |\mathcal{P}|=k}} \mathcal{L}\!\left( f_{\theta}, f_{\theta\setminus \mathcal{P}} \right),$$9 for LLM-Streamline, and $\mathcal{L}(\cdot,\cdot)$0 for ShortGPT (Yun et al., 27 May 2026). On Qwen3-14B, dense accuracy is $\mathcal{L}(\cdot,\cdot)$1; at $\mathcal{L}(\cdot,\cdot)$2, LoRP gives $\mathcal{L}(\cdot,\cdot)$3, compared with $\mathcal{L}(\cdot,\cdot)$4, $\mathcal{L}(\cdot,\cdot)$5, and $\mathcal{L}(\cdot,\cdot)$6 for the three baselines (Yun et al., 27 May 2026).

The appendix reports practical efficiency on Qwen3-14B. Dense latency is $\mathcal{L}(\cdot,\cdot)$7 ms. LoRP gives $\mathcal{L}(\cdot,\cdot)$8 ms at $\mathcal{L}(\cdot,\cdot)$9 (1.36×), $\mathcal{D} = \{x_m\}_{m=1}^M.$0 ms at $\mathcal{D} = \{x_m\}_{m=1}^M.$1 (1.45×), and $\mathcal{D} = \{x_m\}_{m=1}^M.$2 ms at $\mathcal{D} = \{x_m\}_{m=1}^M.$3 (1.56×) (Yun et al., 27 May 2026). Peak GPU memory drops from $\mathcal{D} = \{x_m\}_{m=1}^M.$4 GiB dense to $\mathcal{D} = \{x_m\}_{m=1}^M.$5 GiB (−24.1%), $\mathcal{D} = \{x_m\}_{m=1}^M.$6 GiB (−28.4%), and $\mathcal{D} = \{x_m\}_{m=1}^M.$7 GiB (−32.8%) at the same pruning levels (Yun et al., 27 May 2026).

The strongest empirical pattern is that LoRP’s gains are especially pronounced on Qwen, which the paper identifies as having globally distributed redundancy (Yun et al., 27 May 2026). This directly supports the method’s claim that architecture-sensitive locality modeling matters. The paper also compares LoRP against iterative baselines such as Shortened LLaMA and SLEB, noting that iterative methods can achieve lower perplexity on the calibration-aligned objective, whereas LoRP often preserves higher downstream accuracy, especially on Qwen (Yun et al., 27 May 2026).

5. Relation to adjacent “LoRP-like” methods

Several papers instantiate related ideas under different notions of locality and redundancy. These do not define LoRP in the same sense, but they illuminate the design space.

In 3DGS-SLAM, "Pocket-SLAM: Rendering-Area-Aware Pruning for Memory-Efficient 3DGS-SLAM" (Li et al., 23 Jun 2026) is described as redundancy-aware and weakly locality-aware, but not in the strongest LoRP sense. It prunes Gaussians by their effective rendering-area contribution,

$\mathcal{D} = \{x_m\}_{m=1}^M.$8

and then applies per-tile survivor budgets

$\mathcal{D} = \{x_m\}_{m=1}^M.$9

based on tile-level tracking gradients (Li et al., 23 Jun 2026). Its locality is image-tile locality, not explicit 3D neighborhood locality, and its redundancy score is unary rather than overlap-aware (Li et al., 23 Jun 2026). The paper explicitly notes that a stronger LoRP extension could define local neighborhoods in 3D or projected footprint overlap neighborhoods and estimate each Gaussian’s unique coverage rather than total coverage (Li et al., 23 Jun 2026). This suggests a distinction between region-sensitive budgeting and explicit local mutual redundancy modeling.

In Vision Transformers, "RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT" (Choi et al., 6 Jun 2026) contains a Redundancy-Similarity Aware Pruning stage that is the closest match to a LoRP-like component in that paper. For source token $t$0,

$t$1

and the method prunes the top-$t$2 source tokens with the highest redundancy scores in shallow-to-middle layers (Choi et al., 6 Jun 2026). The paper explicitly states that locality is implicit rather than explicitly geometric: early layers encode local patterns and textures, but the algorithm uses bipartite similarity among token groups rather than spatial neighborhoods (Choi et al., 6 Jun 2026). This makes RAPID redundancy-aware and early-layer local-pattern-aware, but not explicitly neighborhood-based in the spatial sense (Choi et al., 6 Jun 2026).

In VLM pruning, "VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm" (Wu et al., 2 Dec 2025) is much closer to an explicit locality-aware redundancy formulation. It modulates feature-space similarity using nearest-selected spatial distance: $t$3

$t$4

with $t$5, and defines a non-duplication score

$t$6

for near-to-far greedy selection (Wu et al., 2 Dec 2025). The paper explicitly frames this as balancing redundancy reduction with spatial coverage coherence, and reports that replacing its Stage 2 with redundancy-only baselines hurts substantially (Wu et al., 2 Dec 2025). A plausible implication is that VLM-Pruner is closer to a generic “Locality-Aware Redundancy Pruning” concept than LoRP’s layer-axis formulation, but it operates on visual tokens rather than network depth.

These comparisons clarify that LoRP is one member of a broader design family. Shared motifs include moving beyond scalar per-unit heuristics, conditioning pruning on local or regional structure, and defining redundancy in terms of mutual representational overlap rather than isolated saliency. The specific notion of locality, however, changes sharply across domains: depth-axis locality in LLMs (Yun et al., 27 May 2026), image-tile locality in SLAM (Li et al., 23 Jun 2026), shallow-layer local-pattern locality in ViTs (Choi et al., 6 Jun 2026), and 2D or 3D token-grid locality in VLMs (Wu et al., 2 Dec 2025).

6. Limitations, misconceptions, and future directions

LoRP’s limitations are stated explicitly. It is evaluated only on several decoder-only Transformer families, and the RLS-to-$t$7 policy is empirically derived, not theoretically optimal (Yun et al., 27 May 2026). The number of clusters $t$8 is still manually specified through a heuristic thresholding scheme, the method is studied in a no-recovery setting, and calibration uses a fixed C4 setup, leaving stability across domains open (Yun et al., 27 May 2026). The paper also does not provide detailed ablations on alternative similarity metrics, calibration set size sweeps, token-position pooling variants, alternative clustering algorithms, or explicit Stage-1 versus Stage-2 removal comparisons (Yun et al., 27 May 2026).

A common misconception would be to read “locality-aware” as meaning that LoRP prefers neighboring layers by construction. The paper explicitly avoids that assumption. RLS is derived from the global off-diagonal mean of the full similarity matrix, and spectral clustering is chosen precisely because it can group non-adjacent representationally similar layers (Yun et al., 27 May 2026). Locality in LoRP therefore means how localized redundancy happens to be, not a hard contiguity prior.

Another possible misconception is that LoRP performs a purely local layer-importance ranking. It does not. The framework first infers a global representation geometry, then clusters layers, then uses both layer-level and cluster-level redundancy to allocate pruning (Yun et al., 27 May 2026). The pruning objective is therefore relational and architecture-sensitive, even though the final action is simple direct block deletion.

The broader literature suggests several plausible extensions. Pocket-SLAM explicitly identifies the absence of overlap-aware conditional redundancy modeling as a limitation relative to a fuller LoRP formulation (Li et al., 23 Jun 2026). RAPID notes that a more principled criterion for switching from pruning to merging is needed (Choi et al., 6 Jun 2026). VLM-Pruner shows that redundancy-only pruning can become spatially fragmented unless locality enters the scoring rule itself (Wu et al., 2 Dec 2025). Taken together, these works suggest that stronger future LoRP-style methods might combine global redundancy geometry with conditional or overlap-aware redundancy estimates, adaptive locality definitions, or post-pruning recovery.

The clean conclusion is that LoRP, in its explicit 2026 formulation, is a global, architecture-aware, training-free one-shot method for depth pruning that replaces fixed local heuristics with a representation-geometric view of redundancy (Yun et al., 27 May 2026). Its defining technical pieces are pairwise layer-input cosine similarity, the Representation Locality Score,

$t$9

spectral clustering on

$N$00

the intra-cluster layer redundancy score

$N$01

and residual cluster redundancy

$N$02

Empirically, it is especially effective on architectures like Qwen, where redundancy is globally distributed and prior local or contiguous pruning assumptions break down (Yun et al., 27 May 2026).