---
title: Representation Locality Score (RLS)
url: https://www.emergentmind.com/topics/representation-locality-score-rls
type: topic
---

# Representation Locality Score (RLS)

Representation Locality Score (RLS) is a scalar metric introduced in the depth-pruning framework Locality-Aware Redundancy Pruning (LoRP) to characterize how representational redundancy is organized across the depth of a large language model. Rather than treating redundancy as either a purely local property of individual Transformer blocks or as a phenomenon confined to contiguous depth regions, RLS summarizes whether inter-layer similarity is localized or globally distributed across the network. In LoRP, this summary is used before pruning entire Transformer blocks, so that pruning follows the architecture-dependent geometry of layer representations rather than a fixed structural assumption [2605.27786].

## 1. Definition and problem setting

RLS is designed to measure **architecture-dependent representation locality**: whether inter-layer redundancy in a large language model is concentrated in particular depth regions or spread more evenly across many layers [2605.27786]. The motivating observation is that **depth redundancy is not universal**. Some model families exhibit highly similar representations within specific depth neighborhoods, whereas others display similarity patterns that extend across distant layers.

This distinction matters for one-shot depth pruning. If redundancy is **localized**, pruning can be concentrated within those regions with less harm. If redundancy is **globally distributed**, similar layers may be far apart in depth, and pruning should therefore be distributed across multiple representational groups rather than collapsed into a single contiguous block. RLS serves as the summary statistic that differentiates these regimes.

The metric is explicitly positioned against two restrictive assumptions common in prior one-shot pruning methods. One assumption is **local importance / isolated-layer pruning**, in which each layer is judged independently. The other is a **contiguous redundancy assumption**, in which redundancy is presumed to lie in adjacent depth regions. LoRP treats both as incomplete descriptions of depth redundancy, and RLS is introduced so that pruning can instead follow the actual geometry of layer representations [2605.27786].

## 2. Mathematical construction

RLS is derived from **global inter-layer hidden-state similarity** computed on a small calibration set. In the reported setup, the calibration corpus consists of **128 sequences** from **C4**, each of length **2048** [2605.27786]. For each Transformer block $f_l$, a forward hook collects input hidden states. For sample $x_m$ and token position $t$, the hidden state is

$$
\mathbf{h}_{l,m,t} \in \mathbb{R}^d.
$$

These vectors are normalized as

$$
\hat{\mathbf{h}}_{l,m,t} = \frac{\mathbf{h}_{l,m,t}}{\|\mathbf{h}_{l,m,t}\|_2 + \epsilon}.
$$

LoRP then constructs a pairwise layer similarity matrix $\mathbf{S} \in \mathbb{R}^{N \times N}$ by averaging token-wise cosine similarity across the calibration corpus:

$$
S_{ij} = \frac{1}{\sum_{m=1}^{M} T_m} \sum_{m=1}^{M} \sum_{t=1}^{T_m} \hat{\mathbf{h}}_{i,m,t}^{\top} \hat{\mathbf{h}}_{j,m,t},
$$

where $T_m$ is the sequence length for sample $x_m$.

The crucial design choice is that RLS is not based only on nearby layers. Instead, it uses the **global off-diagonal mean similarity**

$$
\bar{S}_{\mathrm{off}} = \frac{2}{N(N-1)} \sum_{i<j} S_{ij},
$$

and defines the Representation Locality Score as

$$
\mathrm{RLS}(\mathbf{S}) = -\log_2 \bar{S}_{\mathrm{off}}.
$$

The paper states that this global averaging is chosen so that the metric does **not assume any particular redundancy pattern** such as contiguity [2605.27786]. In effect, RLS is a log-transformed inverse of the mean pairwise inter-layer similarity. A **higher RLS** indicates that similarity decays faster with depth distance and that redundancy is more localized; a **lower RLS** indicates more globally shared similarity and thus more distributed redundancy.

## 3. Interpretation across model families

The reported similarity heatmaps show contrasting behavior across architectures. **LLaMA, OLMo, and Mistral** exhibit more **localized redundancy**, whereas **Qwen** exhibits more **globally distributed similarity across depth** [2605.27786]. The corresponding RLS values are:

| Model | RLS | LoRP cluster count $K$ |
|---|---:|---:|
| Llama-3.1-8B | 1.149 | 2 |
| OLMo-3-7B | 0.941 | 3 |
| Mistral-Nemo-12B | 0.926 | 3 |
| Qwen3-8B | 0.685 | 4 |
| Qwen3-14B | 0.644 | 4 |

These values instantiate the intended interpretation. Larger RLS values correspond to less global similarity and stronger locality, while smaller RLS values correspond to similarity that is more broadly spread across depth. The paper uses this contrast to argue that pruning should match the architecture’s representational geometry rather than follow a universal pattern [2605.27786].

A plausible implication is that RLS functions as an architecture-level descriptor of redundancy organization, not merely as a heuristic for selecting individual layers. In that sense, it reframes depth pruning as a problem of identifying how redundancy is distributed before deciding where removals should occur.

## 4. Role within the LoRP pruning pipeline

LoRP is a **training-free one-shot depth pruning** method with three main steps: measurement of global inter-layer similarity, clustering of layers by representational similarity, and pruning allocation according to residual intra-cluster redundancy [2605.27786]. RLS is central to the second and third steps because it determines how coarsely or finely the depth dimension should be partitioned.

After computing $\mathbf{S}$, LoRP converts cosine similarity into an affinity matrix,

$$
\mathbf{A} = \frac{\mathbf{S} + 1}{2},
$$

and applies **spectral clustering** to obtain $K$ disjoint clusters,

$$
\mathcal{C} = \{C_1, C_2, \dots, C_K\}.
$$

These are **representational clusters**, not necessarily contiguous depth blocks. The paper specifies an empirical policy linking RLS to cluster granularity:

- $\mathrm{RLS} \ge 1.0 \Rightarrow K=2$
- $0.7 \le \mathrm{RLS} < 1.0 \Rightarrow K=3$
- $\mathrm{RLS} < 0.7 \Rightarrow K=4$

Thus higher RLS leads to coarser or fewer clusters and more concentrated pruning, while lower RLS leads to finer or more numerous clusters and more distributed pruning.

Within a cluster $C_k$, layer redundancy is defined as the average similarity to other layers in the same cluster:

$$
r(l) = \frac{1}{|C_k|-1} \sum_{\substack{m \in C_k \\ m \neq l}} S_{lm}.
$$

Higher $r(l)$ indicates that the layer is more redundant and therefore more removable.

LoRP then performs a **two-stage allocation**. In **coverage-aware initialization**, it picks the most redundant eligible layer from each cluster,

$$
l_k^* = \arg\max_{l \in C_k \setminus \mathcal{B}} r(l),
$$

where $\mathcal{B} = \{1, N\}$ are boundary layers excluded from pruning. The initialization set is

$$
\mathcal{P}_0 = \bigcup_{k=1}^{K} \{l_k^*\}.
$$

The stated purpose is to prevent pruning from collapsing into a single cluster too early.

In **residual redundancy allocation**, the remaining eligible layers in cluster $C_k$ are

$$
\widetilde{C}_k = C_k \setminus \mathcal{P} \setminus \mathcal{B}.
$$

LoRP computes residual cluster redundancy as

$$
\mu_k = \frac{2}{|\widetilde{C}_k|(|\widetilde{C}_k|-1)} \sum_{\substack{i,j \in \widetilde{C}_k \\ i<j}} S_{ij},
$$

and sets

$$
\mu_k = -\infty
$$

when $|\widetilde{C}_k|<2$.

At each step, the method selects the cluster with largest residual redundancy,

$$
k^* = \arg\max_k \mu_k,
$$

then prunes its most redundant remaining layer,

$$
l^* = \arg\max_{l \in \widetilde{C}_{k^*}} r(l),
$$

and updates $\mathcal{P} \leftarrow \mathcal{P} \cup \{l^*\}$ until the pruning budget $k$ is met. In this pipeline, RLS does not directly rank layers; it governs the representational partitioning within which redundancy is subsequently allocated.

## 5. Empirical behavior and validation

The reported experiments evaluate LoRP across five LLM families and indicate that it usually achieves the **lowest average perplexity** among training-free depth pruning methods under **no-recovery settings** [2605.27786]. The gains are described as especially large for models with more globally distributed redundancy, notably **Qwen**.

Examples from the average perplexity results are as follows. For **LLaMA-3.1-8B**, LoRP reports **41.95** at **7/32 pruning** and **151.91** at **9/32 pruning**. For **OLMo-3-7B**, it reports **24.82** at **7/32** and **40.69** at **9/32**. For **Qwen3-8B**, the reported values are **27.92** at **7/36** and **151.64** at **9/36**. For **Qwen3-14B**, they are **37.44** at **11/40** and **76.69** at **13/40**. For **Mistral-Nemo-12B**, the corresponding values are **50.86** at **11/40** and **206.47** at **13/40**.

On **nine zero-shot commonsense benchmarks**, LoRP is reported to generally preserve downstream accuracy better than competing methods. Example average accuracies are **57.21** and **52.06** for **LLaMA-3.1-8B** at **7/32** and **9/32** pruning; **60.38** and **58.20** for **OLMo-3-7B** at **7/32** and **9/32**; **56.65** and **52.43** for **Qwen3-8B** at **7/36** and **9/36**; **52.84** and **49.50** for **Qwen3-14B** at **11/40** and **13/40**; and **55.08** and **51.04** for **Mistral-Nemo-12B** at **11/40** and **13/40** [2605.27786].

The pruning-pattern analysis provides an additional validation of RLS. For **localized models**, selected layer indices tend to concentrate in depth regions. For **distributed models**, pruning is spread across multiple regions. The paper presents this as direct evidence that RLS captures meaningful architecture-level structure. In the appendix, a runtime and memory study on **Qwen3-14B** reports latency decreasing from **676.7 ms** to **432.8 ms**, speedup up to **1.56×**, and peak memory decreasing from **28.12 GiB** to **18.89 GiB** at the most aggressive setting shown.

## 6. Relation to prior one-shot pruning methods

LoRP is contrasted with **ShortGPT**, **Shortened LLaMA**, **LLM-Streamline**, and **LaCo** [2605.27786]. According to the reported comparison, these methods typically rely on local layer scores, isolated removal effects, or a fixed assumption that redundancy is contiguous. RLS is introduced precisely to avoid committing to any of those assumptions in advance.

This distinction also clarifies a common misconception. RLS is not an importance score for individual layers, nor is it a metric restricted to neighboring blocks. It is a global summary of inter-layer similarity structure, computed from off-diagonal hidden-state similarity over the full depth of the model. The paper’s formulation therefore shifts the central question from **which individual layers look unimportant** to **how redundancy is structured across the whole depth of the model**.

The broader significance of RLS within LoRP is that it turns depth pruning into an architecture-adaptive procedure. High RLS implies localized similarity and supports concentrated pruning. Low RLS implies globally distributed similarity and supports pruning spread across multiple clusters. The empirical results are presented as evidence that replacing fixed redundancy assumptions with this architecture-dependent view yields better perplexity and downstream accuracy than prior one-shot depth-pruning methods that depend on local layer scores or contiguous redundancy assumptions [2605.27786].

Source: https://www.emergentmind.com/topics/representation-locality-score-rls