---
title: Empirical Activation Similarity (EAS) Overview
url: https://www.emergentmind.com/topics/empirical-activation-similarity-eas
type: topic
---

# Empirical Activation Similarity (EAS) Overview

Empirical Activation Similarity (EAS) quantifies the statistical alignment or correspondence between high-dimensional activation patterns elicited by different inputs within neural systems, including artificial neural networks and the human brain. EAS metrics have been deployed to measure semantic similarity, to guide model pruning, and to analyze representational specialization across domains and layers. Core instantiations span model-comparison in cognitive neuroscience, gradient-driven attribution in transformer architectures, and angular fidelity loss for deep learning compression. Definitions and protocols vary by field, but EAS typically leverages second-order activation statistics or cosine-based similarity over activation vectors, grounded directly in observed (empirical) activity rather than parametric modeling assumptions.

## 1. Fundamental Formulations of EAS Across Modalities

In large language models (LLMs), EAS formalizes the cosine similarity between “activation vectors” derived from parameterwise gradients of model outputs—measuring which parameters are influential for a particular input. Given a model output functional $D(X, w)$ for input $X$ and parameters $w = (w_1, ..., w_n)$, the per-parameter activation metric is
$$
\mathcal{A}(X, w_i) \approx |w_i\, \partial D(X,w)/\partial w_i|
$$
where $\mathcal{A}(X)$ denotes the $n$-dimensional activation vector. For two inputs $X_1, X_2$, EAS is computed as
$$
\mathrm{EAS}(X_1, X_2) = \frac{\mathcal{A}(X_1) \cdot \mathcal{A}(X_2)}{\|\mathcal{A}(X_1)\|\, \|\mathcal{A}(X_2)\|}
$$
This metric, referred to as LLMDcos, takes values in $[0,1]$ due to nonnegative elements, with unity indicating maximal overlap in activated parameters [2405.17799].

In neural data analysis (e.g., MEG studies), EAS characterizes stimulus similarity via Pearson correlation between empirically reduced activation vectors $x_i(t), x_j(t)$ at time $t$:
$$
RDM_{ij}(t) = 1 - \mathrm{corr}(x_i(t), x_j(t)), \quad 
S_{ij}(t) = 1 - RDM_{ij}(t)
$$
where $S_{ij}(t)$ is the time-resolved empirical activation similarity [1506.02208]. Alternative constructions may use classification-based dissimilarity measures (e.g., $d'$ from decoding analysis) normalized and inverted to produce similarity scores.

For transformer interpretability, gradient × activation saliency maps define tokenwise and word-group activations, enabling EAS-like explanatory matching of words/phrases between text pairs [2208.06612].

## 2. EAS in Model Compression and Pruning

Recent pruning strategies exploit EAS to preserve the angular structure of representations during parameter ablation. In the ACE framework, Empirical Activation Similarity measures the cosine fidelity between unpruned and pruned model activations:
$$
\textrm{EAS} = \frac{1}{N\, d_{\mathsf{out}}} \sum_{t=1}^N \sum_{i=1}^{d_{\mathsf{out}}}
\mathrm{cos}(A_{t,i}, \hat{A}_{t,i})
$$
where $A$ and $\hat{A}$ are dense and pruned layer outputs for an $N$-token batch. The pruning score for each connection combines a weight-magnitude × activation-norm factor (CosP) with an activation-variance factor (VarP):
$$
S_{\mathrm{cos},ij} = |W_{ij}| \cdot \|X_j\|_2 \cdot (1-\cos \hat\theta_{ij})
$$
$$
S_{\mathrm{var},ij} = |W_{ij}| \cdot (\mu_j^2 + v_j + \mu_j + 1)
$$
$$
S_{\mathrm{ACE},ij} = S_{\mathrm{cos},ij} \cdot S_{\mathrm{var},ij}
$$
The ACE algorithm prunes weights ranked by $S_{\mathrm{ACE},ij}$, directly minimizing angular distortion and improving calibration efficiency. Experiments show EAS-informed pruning achieves up to 18% reduction in perplexity and up to 63% reduction in time relative to non-EAS baselines, while requiring as few as 16 tokens of calibration data [2505.21987].

## 3. Layerwise and Domain-Sensitivity Analysis Using EAS

EAS provides a lens onto internal specialization, differentiating “universal encoder” layers (high activation similarity across domains) from deep “expert” layers which activate differently for task-specific or cross-domain inputs [2405.17799]. Empirically:
- Within-domain EAS: High ($\sim 0.9-1.0$) for all layers—parameters consistently co-activated.
- Cross-domain EAS: High for shallow layers then decays to $0.4-0.6$ in deep layers—deep blocks exhibit representational individuality.
- Peak domain-agnosticity in layer 2; maximal specialization in layers 20–30 (for Llama2-7B).

Averaging EAS matrices over datasets recovers an interpretable domain-task similarity structure, accurately reflecting semantic or procedural overlap between benchmarks.

## 4. Empirical Activation Similarity in Cognitive Neuroscience

EAS enables the comparison of neural representations evoked by sensory stimuli. In MEG studies [1506.02208], EAS is calculated as correlation similarity between PCA-reduced activation vectors for each stimulus at each timepoint, yielding dynamic similarity matrices. This approach supports representational similarity analysis (RSA) to compare empirical neural geometry to external models (retinotopic, computational, or perceptual). Key empirical findings:
- Early visual cortex representations align with retinotopic models ($\sim$50–80 ms post-stimulus).
- From $\sim$150 ms, EAS with perceptual-similarity models approaches the empirical noise ceiling.
- EAS provides a metric for empirical quantification of perceptual Gestalts via brain-wide activation patterns.

## 5. Applications: Pruning, Interpretability, Retrieval, Model Calibration

EAS metrics have demonstrable utility:
- **Adaptive Model Pruning**: EAS guides unstructured and semi-structured pruning; layerwise pruning ratios are tuned by observed activation density/sparsity (e.g., densest layers pruned less aggressively) [2405.17799, 2505.21987].
- **Calibration Efficiency**: EAS-based pruning remains robust with very short calibration sequences, supporting rapid compression.
- **Interpretability and Attribution**: EAS-inspired saliency and word-pair matching provide token-level explanations for BERT similarity [2208.06612].
- **Semantic Similarity**: Deep-layer EAS correlates with human judgment on STS-B and SICK, offering embedding-free data relevance signals.
- **Monitoring and Robustness**: Large rotational changes in activation similarity can signal domain shift or calibration drift in deployment [2505.21987].

## 6. Implementation Protocols and Assessment

EAS operationalizes as batch-wise or layer-wise cosine similarity between activation vectors. Variants exist:
- Parameter-space activation statistics (gradient × parameter, [2405.17799])
- Activation vector correlation (MEG, [1506.02208])
- Token/word-level saliency (transformers, [2208.06612])
- Angular deviation between dense and compressed model activations (ACE, [2505.21987])

Calibration batch size ($M$) and sequence length ($N$) are key hyperparameters; practical settings range from $N=16$ to $N=2048$ sequences. Validation is performed by correlating EAS matrices with external similarity labels or task/domain outcomes, using Spearman or Wilcoxon statistics.

## 7. Comparative Table: EAS Usage Across Domains

| Field                      | EAS Formulation                     | Principal Use                |
|----------------------------|-------------------------------------|------------------------------|
| LLMs/Pruning [2505.21987]  | Cosine sim. of dense/pruned acts    | Compression, calibration     |
| LLMs/Interpretability [2405.17799]   | Cosine sim. of activation vectors         | Layer specialization, domain analysis |
| BERT/Interpretation [2208.06612]     | Gradient × activation saliency            | Token/word attribution       |
| Cognitive Neuroscience [1506.02208]  | Corr. similarity over neural acts         | Representational similarity  |

The diversity of EAS instantiations reflects the underlying generality of empirical activation geometry as a unifying framework for measuring representational, functional, and semantic similarity in both artificial and biological systems.

Source: https://www.emergentmind.com/topics/empirical-activation-similarity-eas