Representation Locality Score (RLS)

• Representation Locality Score (RLS) is a scalar metric that quantifies how inter-layer similarity is organized, indicating if redundancy is localized or globally distributed.
• It informs the LoRP pruning pipeline by enabling spectral clustering of Transformer layers based on hidden-state cosine similarity computed from a calibration corpus.
• Empirical results show that higher RLS values reflect concentrated redundancy in specific depth regions, leading to more effective and adaptive depth pruning.

Representation Locality Score (RLS) is a scalar metric introduced in the depth-pruning framework Locality-Aware Redundancy Pruning (LoRP) to characterize how representational redundancy is organized across the depth of a LLM. Rather than treating redundancy as either a purely local property of individual Transformer blocks or as a phenomenon confined to contiguous depth regions, RLS summarizes whether inter-layer similarity is localized or globally distributed across the network. In LoRP, this summary is used before pruning entire Transformer blocks, so that pruning follows the architecture-dependent geometry of layer representations rather than a fixed structural assumption (Yun et al., 27 May 2026).

1. Definition and problem setting

RLS is designed to measure architecture-dependent representation locality: whether inter-layer redundancy in a LLM is concentrated in particular depth regions or spread more evenly across many layers (Yun et al., 27 May 2026). The motivating observation is that depth redundancy is not universal. Some model families exhibit highly similar representations within specific depth neighborhoods, whereas others display similarity patterns that extend across distant layers.

This distinction matters for one-shot depth pruning. If redundancy is localized, pruning can be concentrated within those regions with less harm. If redundancy is globally distributed, similar layers may be far apart in depth, and pruning should therefore be distributed across multiple representational groups rather than collapsed into a single contiguous block. RLS serves as the summary statistic that differentiates these regimes.

The metric is explicitly positioned against two restrictive assumptions common in prior one-shot pruning methods. One assumption is local importance / isolated-layer pruning, in which each layer is judged independently. The other is a contiguous redundancy assumption, in which redundancy is presumed to lie in adjacent depth regions. LoRP treats both as incomplete descriptions of depth redundancy, and RLS is introduced so that pruning can instead follow the actual geometry of layer representations (Yun et al., 27 May 2026).

2. Mathematical construction

RLS is derived from global inter-layer hidden-state similarity computed on a small calibration set. In the reported setup, the calibration corpus consists of 128 sequences from C4, each of length 2048 (Yun et al., 27 May 2026). For each Transformer block $f_l$, a forward hook collects input hidden states. For sample $x_m$ and token position $t$, the hidden state is

$\mathbf{h}_{l,m,t} \in \mathbb{R}^d.$

These vectors are normalized as

$$\hat{\mathbf{h}}_{l,m,t} = \frac{\mathbf{h}_{l,m,t}}{\|\mathbf{h}_{l,m,t}\|_2 + \epsilon}.$$

LoRP then constructs a pairwise layer similarity matrix $\mathbf{S} \in \mathbb{R}^{N \times N}$ by averaging token-wise cosine similarity across the calibration corpus:

$$S_{ij} = \frac{1}{\sum_{m=1}^{M} T_m} \sum_{m=1}^{M} \sum_{t=1}^{T_m} \hat{\mathbf{h}}_{i,m,t}^{\top} \hat{\mathbf{h}}_{j,m,t},$$

where $T_m$ is the sequence length for sample $x_m$.

The crucial design choice is that RLS is not based only on nearby layers. Instead, it uses the global off-diagonal mean similarity

$$\bar{S}_{\mathrm{off}} = \frac{2}{N(N-1)} \sum_{i<j} S_{ij},$$

and defines the Representation Locality Score as

$x_m$0

The paper states that this global averaging is chosen so that the metric does not assume any particular redundancy pattern such as contiguity (Yun et al., 27 May 2026). In effect, RLS is a log-transformed inverse of the mean pairwise inter-layer similarity. A higher RLS indicates that similarity decays faster with depth distance and that redundancy is more localized; a lower RLS indicates more globally shared similarity and thus more distributed redundancy.

3. Interpretation across model families

The reported similarity heatmaps show contrasting behavior across architectures. LLaMA, OLMo, and Mistral exhibit more localized redundancy, whereas Qwen exhibits more globally distributed similarity across depth (Yun et al., 27 May 2026). The corresponding RLS values are:

Model	RLS	LoRP cluster count $x_m$1
Llama-3.1-8B	1.149	2
OLMo-3-7B	0.941	3
Mistral-Nemo-12B	0.926	3
Qwen3-8B	0.685	4
Qwen3-14B	0.644	4

These values instantiate the intended interpretation. Larger RLS values correspond to less global similarity and stronger locality, while smaller RLS values correspond to similarity that is more broadly spread across depth. The paper uses this contrast to argue that pruning should match the architecture’s representational geometry rather than follow a universal pattern (Yun et al., 27 May 2026).

A plausible implication is that RLS functions as an architecture-level descriptor of redundancy organization, not merely as a heuristic for selecting individual layers. In that sense, it reframes depth pruning as a problem of identifying how redundancy is distributed before deciding where removals should occur.

4. Role within the LoRP pruning pipeline

LoRP is a training-free one-shot depth pruning method with three main steps: measurement of global inter-layer similarity, clustering of layers by representational similarity, and pruning allocation according to residual intra-cluster redundancy (Yun et al., 27 May 2026). RLS is central to the second and third steps because it determines how coarsely or finely the depth dimension should be partitioned.

After computing $x_m$2, LoRP converts cosine similarity into an affinity matrix,

$x_m$3

and applies spectral clustering to obtain $x_m$4 disjoint clusters,

$x_m$5

These are representational clusters, not necessarily contiguous depth blocks. The paper specifies an empirical policy linking RLS to cluster granularity:

• $x_m$6
• $x_m$7
• $x_m$8

Thus higher RLS leads to coarser or fewer clusters and more concentrated pruning, while lower RLS leads to finer or more numerous clusters and more distributed pruning.

Within a cluster $x_m$9, layer redundancy is defined as the average similarity to other layers in the same cluster:

$t$0

Higher $t$1 indicates that the layer is more redundant and therefore more removable.

LoRP then performs a two-stage allocation. In coverage-aware initialization, it picks the most redundant eligible layer from each cluster,

$t$2

where $t$3 are boundary layers excluded from pruning. The initialization set is

$t$4

The stated purpose is to prevent pruning from collapsing into a single cluster too early.

In residual redundancy allocation, the remaining eligible layers in cluster $t$5 are

$t$6

LoRP computes residual cluster redundancy as

$t$7

and sets

$t$8

when $t$9.

At each step, the method selects the cluster with largest residual redundancy,

$\mathbf{h}_{l,m,t} \in \mathbb{R}^d.$0

then prunes its most redundant remaining layer,

$\mathbf{h}_{l,m,t} \in \mathbb{R}^d.$1

and updates $\mathbf{h}_{l,m,t} \in \mathbb{R}^d.$2 until the pruning budget $\mathbf{h}_{l,m,t} \in \mathbb{R}^d.$3 is met. In this pipeline, RLS does not directly rank layers; it governs the representational partitioning within which redundancy is subsequently allocated.

5. Empirical behavior and validation

The reported experiments evaluate LoRP across five LLM families and indicate that it usually achieves the lowest average perplexity among training-free depth pruning methods under no-recovery settings (Yun et al., 27 May 2026). The gains are described as especially large for models with more globally distributed redundancy, notably Qwen.

Examples from the average perplexity results are as follows. For LLaMA-3.1-8B, LoRP reports 41.95 at 7/32 pruning and 151.91 at 9/32 pruning. For OLMo-3-7B, it reports 24.82 at 7/32 and 40.69 at 9/32. For Qwen3-8B, the reported values are 27.92 at 7/36 and 151.64 at 9/36. For Qwen3-14B, they are 37.44 at 11/40 and 76.69 at 13/40. For Mistral-Nemo-12B, the corresponding values are 50.86 at 11/40 and 206.47 at 13/40.

On nine zero-shot commonsense benchmarks, LoRP is reported to generally preserve downstream accuracy better than competing methods. Example average accuracies are 57.21 and 52.06 for LLaMA-3.1-8B at 7/32 and 9/32 pruning; 60.38 and 58.20 for OLMo-3-7B at 7/32 and 9/32; 56.65 and 52.43 for Qwen3-8B at 7/36 and 9/36; 52.84 and 49.50 for Qwen3-14B at 11/40 and 13/40; and 55.08 and 51.04 for Mistral-Nemo-12B at 11/40 and 13/40 (Yun et al., 27 May 2026).

The pruning-pattern analysis provides an additional validation of RLS. For localized models, selected layer indices tend to concentrate in depth regions. For distributed models, pruning is spread across multiple regions. The paper presents this as direct evidence that RLS captures meaningful architecture-level structure. In the appendix, a runtime and memory study on Qwen3-14B reports latency decreasing from 676.7 ms to 432.8 ms, speedup up to 1.56×, and peak memory decreasing from 28.12 GiB to 18.89 GiB at the most aggressive setting shown.

6. Relation to prior one-shot pruning methods

LoRP is contrasted with ShortGPT, Shortened LLaMA, LLM-Streamline, and LaCo (Yun et al., 27 May 2026). According to the reported comparison, these methods typically rely on local layer scores, isolated removal effects, or a fixed assumption that redundancy is contiguous. RLS is introduced precisely to avoid committing to any of those assumptions in advance.

This distinction also clarifies a common misconception. RLS is not an importance score for individual layers, nor is it a metric restricted to neighboring blocks. It is a global summary of inter-layer similarity structure, computed from off-diagonal hidden-state similarity over the full depth of the model. The paper’s formulation therefore shifts the central question from which individual layers look unimportant to how redundancy is structured across the whole depth of the model.

The broader significance of RLS within LoRP is that it turns depth pruning into an architecture-adaptive procedure. High RLS implies localized similarity and supports concentrated pruning. Low RLS implies globally distributed similarity and supports pruning spread across multiple clusters. The empirical results are presented as evidence that replacing fixed redundancy assumptions with this architecture-dependent view yields better perplexity and downstream accuracy than prior one-shot depth-pruning methods that depend on local layer scores or contiguous redundancy assumptions (Yun et al., 27 May 2026).