---
title: FAMILIARITY Metric for Zero-Shot NER
url: https://www.emergentmind.com/topics/familiarity-metric
type: topic
---

# FAMILIARITY Metric for Zero-Shot NER

FAMILIARITY is a metric for zero-shot named entity recognition (NER) that estimates the label shift between a synthetic training dataset \(D\) and an evaluation dataset \(Z\) by combining semantic similarity between entity-type labels with the frequency of those labels in the training data [2412.10121]. It was proposed in response to the observation that synthetic datasets used for zero-shot NER often contain entity types that are semantically highly similar to, or even the same as, those in standard evaluation benchmarks, so reported zero-shot \(F_1\) scores can overestimate true zero-shot capability. Within that setting, FAMILIARITY provides a scalar in \([0,1]\) that contextualizes performance by quantifying how much the evaluation label space overlaps with the training label space. The broader literature also uses the term “familiarity” for task-specific scores in speech discrimination, open-set recognition, collaboration networks, psycholinguistics, VR interaction, and adaptive retrieval, but those constructions are formally distinct.

## 1. Problem setting and rationale

Zero-shot NER is defined here as detecting named entities of specific types without any training examples of those types. The specific concern addressed by FAMILIARITY is that current work increasingly relies on large synthetic datasets, automatically generated to cover tens of thousands of distinct entity types, while evaluation is typically performed on standard benchmarks whose labels may already be semantically represented in the synthetic training data [2412.10121].

The metric is therefore intended to quantify label shift, defined in this context as the similarity of labels between training and evaluation datasets. The immediate purpose is evaluative rather than predictive: FAMILIARITY is used to contextualize reported zero-shot NER scores when custom synthetic training datasets are used, and to generate evaluation setups of varying transfer difficulty. A high downstream score is not interpreted identically across all regimes. In particular, the paper states that a high \(F_1\) with \(\mathrm{FAMILIARITY}\approx 0.9\) likely reflects training/evaluation label overlap rather than pure generalization, whereas a high \(F_1\) with \(\mathrm{FAMILIARITY}\approx 0.2\) is evidence of genuine zero-shot success.

The design of the metric reflects two assumptions stated explicitly in the paper. First, semantic proximity between labels matters, so label strings are embedded and compared by cosine similarity. Second, frequency matters, so labels that occur many times in the synthetic training data should contribute more strongly than labels that are present only sparsely.

## 2. Formal definition

Let \(C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}\) be the set of entity-type labels in the synthetic dataset \(D\), and let \(C_Z = \{\tau_1,\tau_2,\dots,\tau_{|Z|}\}\) be the set of entity-type labels in the evaluation dataset \(Z\). Let \(c(\ell)\in\mathbb N\) denote the support count of label \(\ell\in C_D\), let \(\mathrm{emb}(\cdot)\in\mathbb R^d\) be a label-embedding function, and let \(\cos(u,v)=\langle u,v\rangle / (\|u\|\cdot\|v\|)\).

The metric first defines a clipped cosine similarity between an evaluation label \(\tau\in C_Z\) and a training label \(\ell\in C_D\):
\[
\mathrm{clipSim}(\tau,\ell)=\max\bigl(\cos(\mathrm{emb}(\tau),\mathrm{emb}(\ell)),0\bigr).
\]

For a fixed evaluation label \(\tau\), one forms
\[
S_\tau=\{(s_i,c_i): s_i=\mathrm{clipSim}(\tau,\ell_i),\; c_i=c(\ell_i),\; \ell_i\in C_D\}.
\]

The central operation is support-weighted ranking. The paper conceptually repeats each similarity \(s_i\) exactly \(c_i\) times, producing a multiset \(\mathrm{Multiset}_\tau\). Let
\[
s_{\tau,(1)} \ge s_{\tau,(2)} \ge \dots \ge s_{\tau,(K)}
\]
be the \(K\) largest elements of that multiset; if \(\sum_i c_i < K\), the remaining positions are padded with zeros.

These top-\(K\) supported similarities are then aggregated with Zipfian-style weights
\[
p(k)=\frac{(1/k)}{H_K},
\qquad
H_K=\sum_{j=1}^{K}\frac{1}{j}.
\]

The per-label familiarity score is
\[
\mathrm{FAMILIARITY}(\tau)=\sum_{k=1}^{K} p(k)\, s_{\tau,(k)}.
\]

The dataset-level score is the macro-average over evaluation labels:
\[
\mathrm{FAMILIARITY}(D\to Z)=\frac{1}{|C_Z|}\sum_{\tau\in C_Z}\mathrm{FAMILIARITY}(\tau).
\]

Because each \(\mathrm{clipSim}(\tau,\ell)\in[0,1]\) and the weights sum to \(1\), the paper states that
\[
0 \le \mathrm{FAMILIARITY}(D\to Z) \le 1.
\]

## 3. Components and computation

The metric has three explicit components: semantic similarity, frequency support, and weighted aggregation.

| Component | Definition | Function |
|---|---|---|
| Semantic similarity | \(\mathrm{clipSim}(\tau,\ell)=\max(\cos(\mathrm{emb}(\tau),\mathrm{emb}(\ell)),0)\) | Captures label proximity |
| Frequency support | \(c(\ell)\) | Gives more influence to frequently seen training labels |
| Aggregation | Top-\(K\) supported similarities with \(p(k)\propto 1/k\) | Emphasizes strongest overlaps while retaining multiple neighbors |

Operationally, embeddings for all labels in \(C_D\) and \(C_Z\) are precomputed. Zipfian weights \(p(1),\dots,p(K)\) are then computed once using the \(K\)-th harmonic number. For each evaluation label \(\tau\), the method computes clipped cosine similarities to all training labels, sorts those similarities in descending order, and fills a \(K\)-length array by repeating each similarity according to its support count without materializing the full multiset. The per-label score is the weighted sum of that array, and the final dataset-level score is the macro-average over all evaluation labels [2412.10121].

The paper gives explicit computational considerations. Similarity computation is \(O(|C_Z|\cdot |C_D|\cdot d)\). Sorting is \(O(|C_Z|\cdot(|C_D|\log |C_D| + K))\). To avoid unnecessary memory cost, the multiset should not be materialized; instead, a \(K\)-length array is filled sequentially from the sorted \((s,c)\) pairs.

The paper also specifies practical hyperparameters. \(K\) is the number of support examples, with a default of \(1000\). Sentence-transformers, specifically all-mpnet-base-v2, are recommended for label embeddings. FastText is presented as a cheaper option, with the caveat that it may lack nuance.

## 4. Interpretation, bounds, and examples

The intended interpretation is direct. \(\mathrm{FAMILIARITY}\approx 1\) means that for almost every evaluation label \(\tau\), there exists one or more very similar training labels \(\ell\) with ample support; the model has therefore “seen” near-identical types. \(\mathrm{FAMILIARITY}\approx 0\) means that evaluation labels are semantically distant from all training labels or have zero support, corresponding to a true zero-shot scenario [2412.10121].

The paper states several sensitivity properties. Adding new training labels that are unrelated to the evaluation set barely changes FAMILIARITY because the associated similarities are approximately zero. By contrast, increasing the support count \(c(\ell)\) of a training label that is semantically close to some evaluation label raises that label’s sub-score, because the corresponding similarity occupies more of the top-\(K\) supported positions and displaces more distant similarities. Empirically, most synthetic-to-benchmark pairings lie between \(0.3\) and \(0.9\), which the paper characterizes respectively as “large label shift” and “highly overlapping types.”

The toy example in the paper illustrates the mechanics. If \(C_D=\{\mathrm{PERSON}, \mathrm{LOCATION}\}\) with counts \(100\) and \(50\), \(C_Z=\{\mathrm{HUMAN}, \mathrm{PLACE}\}\), \(K=10\), and the cosine similarities are
\[
\cos(\mathrm{HUMAN},\mathrm{PERSON})=0.9,\quad
\cos(\mathrm{HUMAN},\mathrm{LOCATION})=0.3,
\]
\[
\cos(\mathrm{PLACE},\mathrm{PERSON})=0.2,\quad
\cos(\mathrm{PLACE},\mathrm{LOCATION})=0.85,
\]
then the top-10 supported similarities for HUMAN are all \(0.9\), so \(\mathrm{FAMILIARITY}(\mathrm{HUMAN})\approx 0.9\). For PLACE, the top-10 are all \(0.85\), so \(\mathrm{FAMILIARITY}(\mathrm{PLACE})=0.85\). The macro-average is therefore \(0.875\).

A second example in the paper uses real-type labels: if \(D\) contains \(500\) examples of “CORPORATION” and \(30\) of “COMPANY”, and \(Z\) contains “ORGANIZATION”, with embedding similarity about \(0.94\) to both training labels, then the top-\(K\) supported similarities are all \(0.94\), so \(\mathrm{FAMILIARITY}(\mathrm{ORGANIZATION})\approx 0.94\).

## 5. Role in evaluation and split construction

The primary use of FAMILIARITY is evaluative calibration. The paper recommends that when reporting zero-shot NER \(F_1\), researchers should always report the corresponding FAMILIARITY score [2412.10121]. The reason is not that FAMILIARITY replaces task performance, but that it quantifies the transfer difficulty induced by training–evaluation label overlap. Under this view, current evaluation setups are incomplete when they report only \(F_1\) and omit the extent of label shift.

The metric also supports controlled benchmark design. The paper states that researchers can invert the process to create “easy” or “hard” synthetic subsets by computing, for each training label \(\ell\in C_D\), an aggregate similarity such as \(\max_\tau \mathrm{clipSim}(\tau,\ell)\) or an entropy over \(\{\mathrm{clipSim}(\tau,\ell)\}\). Selecting the top \(X\%\) of training labels yields a low-shift split, while selecting the bottom \(X\%\) yields a high-shift split. In practice, the authors constructed three official difficulty tiers—low, medium, and high—by choosing quantiles of the max-similarity distribution.

A common misconception addressed by the paper is that “zero-shot” evaluation is binary: either labels were seen or they were not. FAMILIARITY formalizes the fact that transfer difficulty is gradual. Semantically similar labels with large support counts can make an ostensibly zero-shot evaluation much easier than labels that are both rare and distant in embedding space. This suggests that zero-shot NER results are best interpreted conditionally on quantified label shift rather than on dataset identity alone.

## 6. Related familiarity metrics in other research areas

The broader literature suggests that “familiarity” is a family of task-dependent constructions rather than a single universal metric. Several papers use the term for formally different quantities.

| Domain | Familiarity construction | Role |
|---|---|---|
| Zero-shot NER | Support-weighted clipped label similarity with Zipfian top-\(K\) aggregation [2412.10121] | Quantifies label shift |
| Speech LFE | Relative increase in ABX speaker-discrimination error when train and test languages differ [2206.13415] | Measures language familiarity effect |
| Open-set recognition | Maximum logit or maximum softmax probability [2203.02486] | Scores whether an input is familiar to the classifier |
| Academic team recognition | Counts of direct co-authorship or triangle-motif co-authorship [2204.02667] | Filters cohesive teams |
| Word familiarity | Human rating \(F(w)\in[1,7]\), related to corpus log-frequency and dispersion [1806.03431], [2501.06536] | Psycholinguistic lexical variable |
| VR interaction | Binary self-reported familiarity, with softmax probability proposed as a continuous surrogate [2401.16443] | Detects prior VR experience |
| Personalized retrieval | Mean top-\(K\) similarity plus entropy over probe scores [2603.09250] | Routes between familiarity and recollection retrieval |

In speech processing, the gradual Language Familiarity Effect metric is defined as
\[
\mathrm{LFE}=\frac{S_{\mathrm{diff}}-S_{\mathrm{same}}}{S_{\mathrm{same}}},
\]
where \(S_{\mathrm{same}}\) averages ABX error rates when training and test languages match and \(S_{\mathrm{diff}}\) averages the cross-language conditions. A positive LFE indicates more errors under unfamiliar-language training. Across \(36\) CommonVoice language pairs, the paper reports an overall mean LFE of \(+13.8\%\) with \(95\%\) CI \(2.7\)–\(18.6\%\), and same-family pairs yield mean LFE about \(6.13\%\) versus \(21.46\%\) for different-family pairs [2206.13415]. This is structurally analogous to FAMILIARITY in that both metrics quantify graded transfer difficulty, but the speech metric is defined over ABX discrimination errors rather than label embeddings.

In open-set recognition, familiarity is often operationalized as the maximum logit
\[
F(x)=\max_k \ell_k(x)
\]
or the maximum softmax probability, where smaller values indicate novelty [2203.02486]. A subsequent paper studies adversarial attacks against these familiarity-based scores and defines an adversarial reaction score
\[
R(x)=|S(x)-S(x^{\mathrm{adv}})|.
\]
That work reports a strong positive correlation between the unperturbed maximum logit score and its adversarial reaction, exceeding \(0.8\) in most attack settings, and shows that informed attacks can drive AUROC from about \(0.95\) to about \(0.2\) with FGSM and to about \(0.01\)–\(0.05\) with iterative methods [2311.05006]. Here, familiarity is classifier-internal rather than label-relational.

In graph mining, Yu et al.’s MOTO algorithm defines pairwise familiarity as the number of other team members with whom an author has directly co-authored, and higher-order familiarity as the number of collaborators with whom the author participates in at least one triangle motif [2204.02667]. In psycholinguistics, word familiarity is a human rating on a seven-point scale, and corpus statistics are used as predictors rather than definitions. Tanaka-Ishii and Terada report that correlation between familiarity ratings and log-frequency increases with corpus size and is higher for spoken than written corpora [1806.03431]. Nohejl and Watanabe later show that the logarithm of range is a better predictor than log-frequency across five languages and is also the strongest additional variable when added to log-frequency [2501.06536]. In VR interaction, familiarity is treated as a binary self-report label, and the paper explicitly states that it does not propose a dedicated continuous familiarity score, although softmax probabilities could be used directly as such a score [2401.16443]. In personalized memory retrieval for LLMs, RF-Mem defines a familiarity signal using mean top-\(K\) similarity and the entropy of a temperature-scaled softmax over those similarities; a three-way gating rule then chooses between a direct Familiarity path and a Recollection path [2603.09250].

Taken together, these formulations show that “familiarity” is consistently used to denote prior exposure, overlap, or recognizability, but the mathematical object varies sharply by domain: support-weighted label similarity in zero-shot NER, ABX error shifts in speech, max-logit confidence in open-set recognition, motif counts in collaboration graphs, subjective ratings in psycholinguistics, self-reported labels in VR, and mean-similarity-plus-entropy gating in adaptive retrieval.

Source: https://www.emergentmind.com/topics/familiarity-metric