--- title: Polysemanticity Index (PSI) Explained url: https://www.emergentmind.com/topics/polysemanticity-index-psi type: topic --- # Polysemanticity Index (PSI) Explained Polysemanticity Index (PSI) denotes a family of operational measures for quantifying whether a representation conflates multiple meanings or concepts. The term is not standardized across subfields. In lexical ambiguity detection, PSI has been defined as the residual error of the best single-peaked fit to a radial profile of contextual terms [1905.12065]. In distributional semantics, a closely related detector uses surrounding uniformity to test whether a single embedding vector is consistent with multiple senses [1709.08858]. In explainability and mechanistic interpretability, PSI-like quantities include a normalized Shannon entropy over concept atoms for visual concepts [2503.15234], an average pairwise cosine similarity among multiple textual descriptions of a feature [2506.15538], and a null-calibrated product of geometric separability, class-label alignment, and open-vocabulary distinctness for CNN neurons [2508.16950]. A later critique argues that some purported polysemanticity in language models is substantially attributable to lexical identity confounds, and proposes a lexically adjusted PSI [2604.00443]. ## 1. Scope, sign conventions, and conceptual role Across the cited literature, PSI serves a common purpose—measuring whether one representational unit corresponds to one meaning or several—but the measured object, the mathematical construction, and even the score direction differ. Some formulations treat a high value as evidence for stronger polysemanticity, whereas PRISM’s score increases with semantic coherence and therefore indicates greater monosemanticity [2506.15538]. This variation is central to interpreting published results. | Formulation | Core quantity | High value indicates | |---|---|---| | Lexical PSI | Residual sum-of-squared error of the best unimodal fit to a radial profile | More ambiguity/polysemy | | Surrounding uniformity test | Collinearity of a word vector and its nearest neighbors | More uniformity; lower values suggest polysemy | | Concept Polysemanticity Entropy | Normalized Shannon entropy over concept-atom probabilities | More concept uncertainty/polysemanticity | | PRISM PSI | Mean pairwise cosine similarity among description embeddings | More semantic coherence/monosemanticity | | Null-calibrated CNN PSI | Product $\hat S \times \hat Q \times \hat D$ | Stronger evidence of a polysemantic neuron | The conceptual role also differs by workflow. In lexical settings, PSI is an ambiguity score over words in corpora. In embedding-based word analysis, surrounding uniformity is used as a fast detector to decide which tokens merit more expensive sense analysis. In visual and neural interpretability, PSI-like quantities are used either to quantify uncertainty in a concept representation or to rank neurons and features for further investigation [1709.08858; 2503.15234; 2508.16950]. ## 2. Radial-profile PSI for lexical ambiguity In “Automatic Ambiguity Detection” [1905.12065], the PSI of a target word $w$ is the residual sum-of-squared errors obtained by fitting the word’s radial-distance profile with the best possible single-peaked curve. If the full circle is divided into $n$ angular bins and $x_1,\dots,x_n$ are the smoothed total radial distances in those bins, then \[ \mathrm{PSI}(w) = \min_{1 \le i \le n} \left[ \sum_{j=1}^{i} (\lambda_j - x_j)^2 + \sum_{j=i}^{n} (\mu_j - x_j)^2 \right], \] where $\lambda_1,\dots,\lambda_i$ is the isotonic least-squares fit to $x_1,\dots,x_i$, and $\mu_i,\dots,\mu_n$ is the antitonic least-squares fit to $x_i,\dots,x_n$. The algorithm begins by collecting contextual terms around each occurrence of $w$. With context-window size $k$—typically $k=30$—all tokens $q$ in the $k$ words immediately to the left and right are gathered. A term enters the candidate set $\mathrm{REL}_w$ if it satisfies $\mathrm{FREQ}_w(q) > \mathrm{mincount}$, with $\mathrm{mincount}$ typically $1$, and $\mathrm{FREQ}_w(q)/\mathrm{FREQ}_C(q) > \mathrm{threshold}$, with threshold typically $0.005$. Pairwise distances among candidate terms are then computed using \[ \mathrm{DIST}_w(q,q') = 1 - \sqrt{ \left[\frac{\mathrm{CO}_w(q'|q)}{\mathrm{FREQ}_C(q)}\right] \times \left[\frac{\mathrm{CO}_w(q|q')}{\mathrm{FREQ}_C(q')}\right] }. \] Classical Metric Multidimensional Scaling is applied to this distance matrix to obtain two-dimensional coordinates $(u_q,v_q)$. Those coordinates are converted to radial form via \[ r(q)=\sqrt{u_q^2+v_q^2}, \qquad \theta(q)=\arccos\!\bigl(u_q/r(q)\bigr), \] with $\theta(q)$ mapped into $[0,360)$ by the sign of $v_q$. Angles are quantized into $n$ bins; in the paper, $n=90$, so each bin is $4^\circ$ wide. For each bin, the summed radius is accumulated, optionally with light $3$-bin smoothing, producing the profile $x_1,\dots,x_n$. PSI is then the minimum single-peak fit error across all possible peak locations. The rationale is geometric. A largely unambiguous term produces a single broad hump in the radial histogram, so the unimodal fit incurs little error. A polysemous term yields multiple peaks at different angles, and the enforced single-lobe fit incurs large residuals. The paper’s examples make this explicit: “bass” yields two tight clusters, one musical and one biological, with $\mathrm{PSI}=26.61$, whereas “oxidation” yields a single coherent chemistry cloud with $\mathrm{PSI}=0.23$ [1905.12065]. The reported experimental setting used a 10 million-word Grolier’s Encyclopedia corpus and about 9,000 lower-case words with $50 \le \mathrm{FREQ}_C(w) \le 5{,}000$. Computing PSI for the full list took about 500 CPU-hours on an SGI Origin 200. Among the 25 highest-scoring words were *bill* ($40.08$), *mild* ($32.3$), *seasons* ($31.69$), *inherited* ($29.44$), *moderate* ($28.60$), *bass* ($26.61$), and *tip* ($24.98$). Most matched distinct WordNet entries, though some such as *inherited*, *aromatic*, and *r* were not listed in WordNet, indicating that the procedure can surface unexpected ambiguity [1905.12065]. ## 3. Surrounding-uniformity as a fast polysemy detector for word embeddings The method in “Polysemy Detection in Distributed Representation of Word Sense” [1709.08858] addresses a different question: given a single vector embedding $\vec w$, determine whether it conflates more than one sense. The central quantity is the “surrounding uniformity” (SU), which measures how collinear a word vector is with its nearest neighbors. Let $\vec a_1,\dots,\vec a_N$ be the $N$ nearest-neighbor embeddings of $\vec w$ by smallest angle. Define \[ \vec s(\vec w)=\vec w+\sum_{i=1}^{N}\vec a_i \] and \[ \mathrm{SU}(\vec w) = \frac{\|\vec s(\vec w)\|}{\|\vec w\|+\sum_{i=1}^{N}\|\vec a_i\|} \quad \in [0,1]. \] If all vectors point in the same direction, $\mathrm{SU}=1$; if they fan out widely, $\mathrm{SU}$ is much smaller. The test compares $\mathrm{SU}(\vec w)$ with the SU values of its neighbors. If \[ m=\frac{1}{N}\sum_{i=1}^{N}\mathrm{SU}(\vec a_i), \qquad \sigma= \sqrt{ \frac{1}{N-1} \sum_{i=1}^{N} \bigl(\mathrm{SU}(\vec a_i)-m\bigr)^2 }, \] then $\vec w$ is declared polysemic when \[ \mathrm{SU}(\vec w) < m-k\sigma, \] with $k=3$, yielding a $3\sigma$ outlier rule. The algorithmic recipe is explicit. A corpus is lowercased, word2vec is trained with 200 dimensions, default window, and skip-gram, and the top `limit` words by frequency are retained as “stable words.” For each stable word, the top $N$ nearest stable neighbors are fetched by cosine or angle, SU is computed for the target and each neighbor, and the outlier rule is applied. If any neighbor’s SU is undefined because it lacks $N$ neighbors, the target is skipped. The paper highlights `limit`, $N$, and $k$ as the only tuning knobs, and gives complexity per word as naive $O(\mathtt{limit}\times d)$ for k-NN search plus $O(Nd)$ for SU computation, with overall $O(\mathtt{limit}\times(\mathrm{kNN}+Nd))$ [1709.08858]. The worked example is “may,” whose neighbors are *can* ($0.9252$), *should* ($0.9232$), *might* ($0.9179$), and *will* ($0.9266$). The neighbor mean is $m=0.9232$, the sample standard deviation is $\sigma \approx 0.0038$, and the threshold is $m-3\sigma=0.9118$. Since $\mathrm{SU}(\text{may})=0.8917 < 0.9118$, the word is flagged as polysemic [1709.08858]. On the top 1,000 stable words, 127 were undetermined, 33 passed the polysemy test, and 840 were labeled monosemic. A human evaluation on 24 words produced the confusion matrix with 19 mono/mono, 1 mono/poly, 1 poly/mono, and 3 poly/poly, and a $\chi^2$ test with Yates’ correction showed significance at $\alpha=0.05$ [1709.08858]. The caveats are substantial. Lowercasing can artificially introduce polysemy, as with *may* as month versus modal. Rare senses may be missed, as in *march* as verb or *august* as adjective. The choice $N=4,k=3$ is minimal for a statistical test; larger $N$ may stabilize estimates but risks admitting weakly related neighbors. SU also correlates with part-of-speech, with prepositions having low SU and proper nouns high SU, so absolute thresholds can mix POS effects with polysemy [1709.08858]. ## 4. Entropy-based PSI in visual concept explainability In “CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantification” [2503.15234], the general notion of a Polysemanticity Index is instantiated as Concept Polysemanticity Entropy (CPE), a normalized Shannon entropy over concept atoms associated with a visual concept. A single visual concept $s$ is represented by $N$ image patches. A disentanglement procedure $g$ yields concept atoms, which are clustered into a final atom set \[ \mathcal{A}^\ast=\{a^1,\ldots,a^{P^\ast}\}. \] With $Q$ atoms proposed per patch before clustering and $N$ patches per concept, padding is introduced as \[ \mathrm{Pad}=\max(0,N-P^\ast), \] and the padded probability of atom $i$ is \[ p_i=\frac{\mathrm{Num}_i}{Q\cdot N+\mathrm{Pad}} \qquad (i=1,\ldots,P^\ast+\mathrm{Pad}), \] where $\mathrm{Num}_i$ is the total count of atom $a^i$ across patches. The entropy score is \[ H_j^\ast = \frac{ -\sum_{i=1}^{P^\ast+\mathrm{Pad}} p_i \log p_i }{ \log(P^\ast+\mathrm{Pad}) }, \qquad 0 \le H_j^\ast \le 1. \] Layer- and model-level averages are then \[ H_l^\ast=\frac{1}{d_l}\sum_{j=1}^{d_l}H_j^\ast, \qquad H_{\mathcal M}^\ast=\frac{1}{L}\sum_{l=1}^{L}H_l^\ast. \] The interpretation is distributional. The term $\mathrm{Num}_i/(QN+\mathrm{Pad})$ turns a visual concept into a discrete probability distribution over semantic atoms. The Shannon entropy measures the uncertainty of that distribution. A broad, flat distribution yields high entropy and thus high polysemanticity; a sharply peaked distribution yields low entropy and near-monosemanticity. The normalization by $\log(P^\ast+\mathrm{Pad})$ makes the score comparable across concepts with different atom inventories [2503.15234]. The workflow has five stages: disentangle, cluster, count, pad, and compute. The padding heuristic is introduced so that a concept with very few clustered atoms does not appear maximally polysemantic merely because the support size is too small. The paper explicitly contrasts this with a naive PSI that would simply count the number of atoms $P^\ast$; CPE instead weights atoms by empirical frequency, aggregates them via Shannon entropy, and normalizes to $[0,1]$ [2503.15234]. The empirical illustrations are reported on ResNet152. In Stage 4, channel 75 had $H^\ast=0.81$, described as moderately polysemantic, whereas channel 697 scored $0.95$, described as very polysemantic. Stage 3 typically had the highest averaged CPE, while Stages 1 and 4 had lower CPE. The paper also reports that deeper or more complex networks tend to have higher global $H^\ast_{\mathcal M}$, that human agreement with CPE ordering on 300 VC-pairs was 75%, and that no hard threshold is prescribed, though in practice $H^\ast>0.8$ is regarded as strongly polysemantic and $H^\ast<0.2$ as nearly monosemantic [2503.15234]. The reported caveats are methodologically important. CPE depends on disentanglement quality from the LVLM and clustering quality from the NLI model. The padding heuristic is explicitly described as ad hoc. Shannon entropy also treats all atom differences equally, even when two atoms are semantically close. Suggested extensions include semantic-distance-weighted entropy, Multi-Information decompositions, conditional CPE tied to downstream relevance, and ensemble PSI across different concept-extraction methods [2503.15234]. ## 5. Feature-level PSI in mechanistic interpretability PRISM, introduced in “Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework” [2506.15538], defines PSI directly from multiple textual descriptions of a feature. Let feature $f_{\ell,i}$ have $k$ descriptions $\{d_1,\dots,d_k\}$, and let a sentence encoder $e:\mathcal S \to \mathbb R^T$ produce embeddings $s_j=e(d_j)$. PRISM computes all pairwise cosine similarities \[ \cos(s_p,s_q) = \frac{ \sum_{t=1}^{T}s_{p,t}s_{q,t} }{ \sqrt{\sum_{t=1}^{T}s_{p,t}^2}\; \sqrt{\sum_{t=1}^{T}s_{q,t}^2} }, \] and defines \[ \mathrm{PSI}(f_{\ell,i}) = \frac{2}{k(k-1)} \sum_{1\le p\bar M_{SYN}$ with Wilcoxon $p<10^{-3}$ under Holm-Bonferroni correction. Lexical contribution ratios start very high in early layers, for example $0.74$–$0.91$ at layer $0$, decline with depth to $0.25$–$0.45$ in mid-to-late layers, but never fall below zero. In sparse autoencoders, $18$–$36\%$ of active features conflate multiple senses of the same word. On GPT-2, the correction reclassifies an extra $6.5$–$10.4\%$ of top-polysemantic neurons as purely lexical, of which $93$–$94\%$ are confirmed form detectors with $\mathrm{SSI}<0.5$. Filtering out lexical-only activations improves downstream tasks: for CoarseWSD-20, top sense-selective neurons yield $89.8\%$ accuracy versus $83.6\%$ for sense-blind neurons, and in ROME editing selectivity rises from $0.519$ to $0.540$ with $p=0.002$ [2604.00443]. Other limitations are formulation-specific. The surrounding-uniformity test is sensitive to embedding method, lowercasing, and part-of-speech, and may miss rare senses [1709.08858]. CPE inherits biases from LVLM-based atom extraction and NLI-based clustering, and its padding rule is ad hoc [2503.15234]. PRISM’s score depends on the number of descriptions and the sentence encoder [2506.15538]. The null-calibrated CNN PSI depends on the text encoder and prompt set, and the paper reports only moderate correlation, $\rho \approx 0.21$, when swapping CLIP text heads for raw $D$ scores [2508.16950]. Taken together, these results suggest that PSI is best understood as a task-specific operational statistic rather than a single canonical property. In lexical ambiguity detection, the central question is whether contextual geometry departs from unimodality. In word embeddings, the question is whether a target vector and its nearest neighbors fail to form a tight cone. In explainability, the question may be uncertainty over concept atoms, dispersion among textual descriptions, or null-calibrated evidence that top activations decompose into semantically distinct prototypes. The practical implication is that PSI values are only comparable within a shared definition, preprocessing pipeline, and sign convention. Source: https://www.emergentmind.com/topics/polysemanticity-index-psi