FAMILIARITY Metric for Zero-Shot NER

• FAMILIARITY is a metric that quantifies label shift in zero-shot NER by measuring semantic closeness and frequency support between synthetic training and evaluation labels.
• It computes clipped cosine similarities between label embeddings and aggregates the top-K supported similarities using Zipfian weights.
• A higher FAMILIARITY score indicates significant overlap between training and evaluation datasets, highlighting potential bias in zero-shot performance.

FAMILIARITY is a metric for zero-shot named entity recognition (NER) that estimates the label shift between a synthetic training dataset $D$ and an evaluation dataset $Z$ by combining semantic similarity between entity-type labels with the frequency of those labels in the training data (Golde et al., 2024). It was proposed in response to the observation that synthetic datasets used for zero-shot NER often contain entity types that are semantically highly similar to, or even the same as, those in standard evaluation benchmarks, so reported zero-shot $F_1$ scores can overestimate true zero-shot capability. Within that setting, FAMILIARITY provides a scalar in $[0,1]$ that contextualizes performance by quantifying how much the evaluation label space overlaps with the training label space. The broader literature also uses the term “familiarity” for task-specific scores in speech discrimination, open-set recognition, collaboration networks, psycholinguistics, VR interaction, and adaptive retrieval, but those constructions are formally distinct.

1. Problem setting and rationale

Zero-shot NER is defined here as detecting named entities of specific types without any training examples of those types. The specific concern addressed by FAMILIARITY is that current work increasingly relies on large synthetic datasets, automatically generated to cover tens of thousands of distinct entity types, while evaluation is typically performed on standard benchmarks whose labels may already be semantically represented in the synthetic training data (Golde et al., 2024).

The metric is therefore intended to quantify label shift, defined in this context as the similarity of labels between training and evaluation datasets. The immediate purpose is evaluative rather than predictive: FAMILIARITY is used to contextualize reported zero-shot NER scores when custom synthetic training datasets are used, and to generate evaluation setups of varying transfer difficulty. A high downstream score is not interpreted identically across all regimes. In particular, the paper states that a high $F_1$ with $\mathrm{FAMILIARITY}\approx 0.9$ likely reflects training/evaluation label overlap rather than pure generalization, whereas a high $F_1$ with $\mathrm{FAMILIARITY}\approx 0.2$ is evidence of genuine zero-shot success.

The design of the metric reflects two assumptions stated explicitly in the paper. First, semantic proximity between labels matters, so label strings are embedded and compared by cosine similarity. Second, frequency matters, so labels that occur many times in the synthetic training data should contribute more strongly than labels that are present only sparsely.

2. Formal definition

Let $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$ be the set of entity-type labels in the synthetic dataset $D$, and let $Z$0 be the set of entity-type labels in the evaluation dataset $Z$1. Let $Z$2 denote the support count of label $Z$3, let $Z$4 be a label-embedding function, and let $Z$5.

The metric first defines a clipped cosine similarity between an evaluation label $Z$6 and a training label $Z$7: $Z$8

For a fixed evaluation label $Z$9, one forms

$F_1$0

The central operation is support-weighted ranking. The paper conceptually repeats each similarity $F_1$1 exactly $F_1$2 times, producing a multiset $F_1$3. Let

$F_1$4

be the $F_1$5 largest elements of that multiset; if $F_1$6, the remaining positions are padded with zeros.

These top-$F_1$7 supported similarities are then aggregated with Zipfian-style weights

$F_1$8

The per-label familiarity score is

$F_1$9

The dataset-level score is the macro-average over evaluation labels: $[0,1]$0

Because each $[0,1]$1 and the weights sum to $[0,1]$2, the paper states that

$[0,1]$3

3. Components and computation

The metric has three explicit components: semantic similarity, frequency support, and weighted aggregation.

Component	Definition	Function
Semantic similarity	$[0,1]$4	Captures label proximity
Frequency support	$[0,1]$5	Gives more influence to frequently seen training labels
Aggregation	Top-$[0,1]$6 supported similarities with $[0,1]$7	Emphasizes strongest overlaps while retaining multiple neighbors

Operationally, embeddings for all labels in $[0,1]$8 and $[0,1]$9 are precomputed. Zipfian weights $F_1$0 are then computed once using the $F_1$1-th harmonic number. For each evaluation label $F_1$2, the method computes clipped cosine similarities to all training labels, sorts those similarities in descending order, and fills a $F_1$3-length array by repeating each similarity according to its support count without materializing the full multiset. The per-label score is the weighted sum of that array, and the final dataset-level score is the macro-average over all evaluation labels (Golde et al., 2024).

The paper gives explicit computational considerations. Similarity computation is $F_1$4. Sorting is $F_1$5. To avoid unnecessary memory cost, the multiset should not be materialized; instead, a $F_1$6-length array is filled sequentially from the sorted $F_1$7 pairs.

The paper also specifies practical hyperparameters. $F_1$8 is the number of support examples, with a default of $F_1$9. Sentence-transformers, specifically all-mpnet-base-v2, are recommended for label embeddings. FastText is presented as a cheaper option, with the caveat that it may lack nuance.

4. Interpretation, bounds, and examples

The intended interpretation is direct. $\mathrm{FAMILIARITY}\approx 0.9$0 means that for almost every evaluation label $\mathrm{FAMILIARITY}\approx 0.9$1, there exists one or more very similar training labels $\mathrm{FAMILIARITY}\approx 0.9$2 with ample support; the model has therefore “seen” near-identical types. $\mathrm{FAMILIARITY}\approx 0.9$3 means that evaluation labels are semantically distant from all training labels or have zero support, corresponding to a true zero-shot scenario (Golde et al., 2024).

The paper states several sensitivity properties. Adding new training labels that are unrelated to the evaluation set barely changes FAMILIARITY because the associated similarities are approximately zero. By contrast, increasing the support count $\mathrm{FAMILIARITY}\approx 0.9$4 of a training label that is semantically close to some evaluation label raises that label’s sub-score, because the corresponding similarity occupies more of the top-$\mathrm{FAMILIARITY}\approx 0.9$5 supported positions and displaces more distant similarities. Empirically, most synthetic-to-benchmark pairings lie between $\mathrm{FAMILIARITY}\approx 0.9$6 and $\mathrm{FAMILIARITY}\approx 0.9$7, which the paper characterizes respectively as “large label shift” and “highly overlapping types.”

The toy example in the paper illustrates the mechanics. If $\mathrm{FAMILIARITY}\approx 0.9$8 with counts $\mathrm{FAMILIARITY}\approx 0.9$9 and $F_1$0, $F_1$1, $F_1$2, and the cosine similarities are

$F_1$3

$F_1$4

then the top-10 supported similarities for HUMAN are all $F_1$5, so $F_1$6. For PLACE, the top-10 are all $F_1$7, so $F_1$8. The macro-average is therefore $F_1$9.

A second example in the paper uses real-type labels: if $\mathrm{FAMILIARITY}\approx 0.2$0 contains $\mathrm{FAMILIARITY}\approx 0.2$1 examples of “CORPORATION” and $\mathrm{FAMILIARITY}\approx 0.2$2 of “COMPANY”, and $\mathrm{FAMILIARITY}\approx 0.2$3 contains “ORGANIZATION”, with embedding similarity about $\mathrm{FAMILIARITY}\approx 0.2$4 to both training labels, then the top-$\mathrm{FAMILIARITY}\approx 0.2$5 supported similarities are all $\mathrm{FAMILIARITY}\approx 0.2$6, so $\mathrm{FAMILIARITY}\approx 0.2$7.

5. Role in evaluation and split construction

The primary use of FAMILIARITY is evaluative calibration. The paper recommends that when reporting zero-shot NER $\mathrm{FAMILIARITY}\approx 0.2$8, researchers should always report the corresponding FAMILIARITY score (Golde et al., 2024). The reason is not that FAMILIARITY replaces task performance, but that it quantifies the transfer difficulty induced by training–evaluation label overlap. Under this view, current evaluation setups are incomplete when they report only $\mathrm{FAMILIARITY}\approx 0.2$9 and omit the extent of label shift.

The metric also supports controlled benchmark design. The paper states that researchers can invert the process to create “easy” or “hard” synthetic subsets by computing, for each training label $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$0, an aggregate similarity such as $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$1 or an entropy over $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$2. Selecting the top $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$3 of training labels yields a low-shift split, while selecting the bottom $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$4 yields a high-shift split. In practice, the authors constructed three official difficulty tiers—low, medium, and high—by choosing quantiles of the max-similarity distribution.

A common misconception addressed by the paper is that “zero-shot” evaluation is binary: either labels were seen or they were not. FAMILIARITY formalizes the fact that transfer difficulty is gradual. Semantically similar labels with large support counts can make an ostensibly zero-shot evaluation much easier than labels that are both rare and distant in embedding space. This suggests that zero-shot NER results are best interpreted conditionally on quantified label shift rather than on dataset identity alone.

6. Related familiarity metrics in other research areas

The broader literature suggests that “familiarity” is a family of task-dependent constructions rather than a single universal metric. Several papers use the term for formally different quantities.

Domain	Familiarity construction	Role
Zero-shot NER	Support-weighted clipped label similarity with Zipfian top-$C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$5 aggregation (Golde et al., 2024)	Quantifies label shift
Speech LFE	Relative increase in ABX speaker-discrimination error when train and test languages differ (Seyssel et al., 2022)	Measures language familiarity effect
Open-set recognition	Maximum logit or maximum softmax probability (Dietterich et al., 2022)	Scores whether an input is familiar to the classifier
Academic team recognition	Counts of direct co-authorship or triangle-motif co-authorship (Yu et al., 2022)	Filters cohesive teams
Word familiarity	Human rating $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$6, related to corpus log-frequency and dispersion (Tanaka-Ishii et al., 2018, Nohejl et al., 11 Jan 2025)	Psycholinguistic lexical variable
VR interaction	Binary self-reported familiarity, with softmax probability proposed as a continuous surrogate (Li et al., 2024)	Detects prior VR experience
Personalized retrieval	Mean top-$C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$7 similarity plus entropy over probe scores (Zhang et al., 10 Mar 2026)	Routes between familiarity and recollection retrieval

In speech processing, the gradual Language Familiarity Effect metric is defined as

$C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$8

where $C_D = \{\ell_1,\ell_2,\dots,\ell_{|D|}\}$9 averages ABX error rates when training and test languages match and $D$0 averages the cross-language conditions. A positive LFE indicates more errors under unfamiliar-language training. Across $D$1 CommonVoice language pairs, the paper reports an overall mean LFE of $D$2 with $D$3 CI $D$4–$D$5, and same-family pairs yield mean LFE about $D$6 versus $D$7 for different-family pairs (Seyssel et al., 2022). This is structurally analogous to FAMILIARITY in that both metrics quantify graded transfer difficulty, but the speech metric is defined over ABX discrimination errors rather than label embeddings.

In open-set recognition, familiarity is often operationalized as the maximum logit

$D$8

or the maximum softmax probability, where smaller values indicate novelty (Dietterich et al., 2022). A subsequent paper studies adversarial attacks against these familiarity-based scores and defines an adversarial reaction score

$D$9

That work reports a strong positive correlation between the unperturbed maximum logit score and its adversarial reaction, exceeding $Z$00 in most attack settings, and shows that informed attacks can drive AUROC from about $Z$01 to about $Z$02 with FGSM and to about $Z$03–$Z$04 with iterative methods (Enevoldsen et al., 2023). Here, familiarity is classifier-internal rather than label-relational.

In graph mining, Yu et al.’s MOTO algorithm defines pairwise familiarity as the number of other team members with whom an author has directly co-authored, and higher-order familiarity as the number of collaborators with whom the author participates in at least one triangle motif (Yu et al., 2022). In psycholinguistics, word familiarity is a human rating on a seven-point scale, and corpus statistics are used as predictors rather than definitions. Tanaka-Ishii and Terada report that correlation between familiarity ratings and log-frequency increases with corpus size and is higher for spoken than written corpora (Tanaka-Ishii et al., 2018). Nohejl and Watanabe later show that the logarithm of range is a better predictor than log-frequency across five languages and is also the strongest additional variable when added to log-frequency (Nohejl et al., 11 Jan 2025). In VR interaction, familiarity is treated as a binary self-report label, and the paper explicitly states that it does not propose a dedicated continuous familiarity score, although softmax probabilities could be used directly as such a score (Li et al., 2024). In personalized memory retrieval for LLMs, RF-Mem defines a familiarity signal using mean top-$Z$05 similarity and the entropy of a temperature-scaled softmax over those similarities; a three-way gating rule then chooses between a direct Familiarity path and a Recollection path (Zhang et al., 10 Mar 2026).

Taken together, these formulations show that “familiarity” is consistently used to denote prior exposure, overlap, or recognizability, but the mathematical object varies sharply by domain: support-weighted label similarity in zero-shot NER, ABX error shifts in speech, max-logit confidence in open-set recognition, motif counts in collaboration graphs, subjective ratings in psycholinguistics, self-reported labels in VR, and mean-similarity-plus-entropy gating in adaptive retrieval.