Definition-Specific Familiarity (DSF)

• DSF is a diagnostic metric that measures how well a model’s implicit concept understanding matches the explicit task definition.
• It employs cosine similarity across multiple sentence encoders to compare elicited concept descriptions with dataset definitions.
• Empirical results demonstrate that higher DSF correlates with improved zero-shot performance and accurate concept application in diverse domains.

Definition-Specific Familiarity (DSF) is a concept-level diagnostic for quantifying how closely a LLM’s internalized understanding of a target phenomenon aligns with the specific task definition a user intends to apply. Introduced in the study "On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance" (Casanova et al., 30 May 2026), DSF is designed for zero-shot annotation and LLM-as-a-judge settings, where the central issue is not whether a model has seen similar text, but whether its implicit decision boundary matches the dataset’s operational definition. In that framing, DSF serves as a label-free proxy for definition alignment and is explicitly contrasted with text-level contamination or memorization metrics.

1. Conceptual basis and motivation

DSF is motivated by the distinction between a model’s internal concept and a dataset’s written definition. Alignment is high when the model’s implicit decision rule for a concept such as toxicity places the positive–negative boundary where the task requires; alignment is low when the model applies a systematically different boundary, even if it has encountered similar texts before (Casanova et al., 30 May 2026). The diagnostic is therefore definition-specific rather than corpus-specific.

The reported motivation is empirical as well as conceptual. In the study’s annotation setting, prompting alone often could not override entrenched priors: nearly two-thirds of zero-shot errors persisted even after adding aligned definitions and examples, and the overall rescue rate, defined as the fraction of initial errors corrected by prompting, was 34.8% (Casanova et al., 30 May 2026). This suggests that prompt-following is constrained by model-internalized priors about what the target concept means.

The same study further argues that definition alignment can matter more than model identity in some settings. Definition choice produced larger accuracy swings than model choice in some datasets, at approximately $17\%$ versus approximately $5\%$, respectively. A plausible implication is that DSF captures a first-order property of annotation reliability: whether the model is already predisposed to interpret the concept in the intended way.

2. Formal definition

Let $m$ index models and $d$ index datasets or tasks. Let $g_d \in T$ denote the dataset’s ground-truth definition text, and let $e_{m,d} \in T$ denote the model’s elicited self-description of the target concept, such as a response to “In your own words, what makes content toxic?” Sentence encoders are written as $\phi_k: T \to \mathbb{R}^{p_k}$, with $k=1,\dots,K$. In the study, $K=6$: MiniLM, MPNet, BGE-large, E5-large, Instructor-large, and OpenAI text-embedding-3-small (Casanova et al., 30 May 2026).

Cosine similarity is defined as

$$\operatorname{cos}(u,v)=\frac{u^\top v}{\|u\|_2\,\|v\|_2}.$$

The per-encoder DSF is

$5\%$0

The consensus DSF used in the paper is the unweighted mean across encoders:

$5\%$1

The symbols have fixed roles: $5\%$2 is the model index, $5\%$3 the dataset or task index, $5\%$4 the dataset definition, $5\%$5 the elicited concept description, $5\%$6 the $5\%$7-th sentence encoder, $5\%$8 the encoder-specific cosine similarity, and $5\%$9 the consensus score. Because the metric compares concept descriptions to definitions rather than model outputs to labels, no labeled examples are required for its computation (Casanova et al., 30 May 2026).

The study also explored weaker variants. “Positive DSF” aligned only the positive class definition via multiple prompt styles, while “Enhanced DSF” averaged alignment to positive and negative definitions and added domain qualifiers such as “in gaming chat.” Both variants showed weaker associations with performance than the consensus formulation.

3. Estimation procedure and operationalization

The inputs required for DSF are the task definition, expressed as the full text used to operationalize labels or rubric criteria, and the model under consideration. The procedure begins by eliciting the model’s concept description with a neutral, non-leading prompt. The study gives the example: “You are documenting your internal understanding of a labeling concept. Concept: [Concept Name]. Describe, in your own words, what content qualifies as this concept.” The dataset’s own definition should not be included in this elicitation prompt, because doing so can induce parroting and trivially inflate similarity. All inference is conducted at temperature $m$0 for reproducibility (Casanova et al., 30 May 2026).

Operationally, the concept description and the dataset definition are embedded by six diverse encoders: all-MiniLM-L6-v2, all-mpnet-base-v2, BAAI/bge-large-en-v1.5, intfloat/e5-large-v2, hkunlp/instructor-large, and text-embedding-3-small. DSF is then computed as the unweighted mean of the six cosine similarities. The recommended use is comparative rather than threshold-based: rank candidate models by DSF for a task, then select high-DSF candidates for pilot annotation or judging (Casanova et al., 30 May 2026).

The reported practical guidance emphasizes several constraints. The elicitation prompt should remain definition-free; DSF should be computed across multiple embedding spaces, preferably at least $m$1–$m$2 encoders; and absolute cutoffs should be avoided because they depend on encoder choice and domain. The study also recommends stress-testing several plausible phrasings of the task definition and observing DSF sensitivity across the models intended for deployment. This suggests a dual use: model selection and definition selection.

4. Experimental setting and statistical evidence

The empirical evaluation used toxicity, hate, and offense datasets across multiple domains, with 1,000 sampled instances per model–dataset pair (Casanova et al., 30 May 2026).

Dataset	Domain	Details
Twitter Hate	Social Media	24,783; 5.8% positive
OLID	Social Media	14,100; 33.2% positive
GameTox	Gaming	53,000; 42.5% positive
Fox News	News	1,528; 28.5% positive
Jigsaw Toxic Comments	Forum	159,571; 9.6% positive

Supplementary datasets were Jigsaw Unintended Bias, Irony, and Subjectivity, used for robustness and generalization. The evaluated instruction-tuned models included dense models—Llama-3.1-8B, Llama-3.1-70B, Llama-3.3-70B, Mistral-7B, Mistral-Small-24B, and Qwen-2.5-72B—mixture-of-experts models—Mixtral-8x7B and DeepSeek-V3—and the proprietary model GPT-4o-mini. Prompts were tested under zero-shot, aligned definition, few-shot with four balanced examples, few-shot plus definition, six misaligned definition swaps, and DSPy-optimized variants. Models output both PREDICTION and CONFIDENCE on a $m$3–$m$4 scale (Casanova et al., 30 May 2026).

The central statistical result is that DSF is positively associated with zero-shot accuracy after controlling for dataset-level variation. Across $m$5 model–dataset pairs, the consensus DSF achieved a partial correlation of $m$6 with $m$7. The association persisted after controlling for text complexity measured by negative log-likelihood. Per-encoder partial correlations ranged from $m$8 to $m$9, and pairwise DSF vectors across encoders were highly concordant, with Pearson $d$0. In few-shot settings the association strengthened to partial $d$1, whereas under aligned definitions it attenuated to partial $d$2 with $d$3, consistent with the interpretation that explicit instructions compress between-model variation (Casanova et al., 30 May 2026).

Robustness claims extend beyond toxicity. On irony and subjectivity tasks, DSF remained predictive, with partial $d$4. The study therefore presents DSF not as a safety-specific heuristic, but as a more general measure of concept–definition fit.

5. Relation to memorization and decision stickiness

A central contrast in the study is between concept-boundary alignment and text-level memorization. Memorization was measured by prompting a model with the first $d$5 prefix of a text, generating a continuation, and comparing that continuation to the held-out $d$6 suffix using ROUGE-L F1, BERTScore F1 with a DeBERTa-XL encoder, and embedding cosine similarity. After controlling for dataset, none of these metrics showed a positive association with accuracy: ROUGE-L had partial $d$7, BERTScore partial $d$8, and embedding similarity partial $d$9 (Casanova et al., 30 May 2026).

A common misconception is that annotation performance in zero-shot settings mainly reflects contamination or recovery of memorized dataset text. The reported results do not support that interpretation. The negative direction holds across lexical and semantic metrics, from ROUGE-L through BERTScore to embedding similarity, while DSF remains positively associated with performance. The study’s interpretation is that memorization reflects text reproduction capability rather than correct placement of the task boundary.

The same analysis introduces “decision stickiness,” defined as the tendency for high-confidence zero-shot errors to resist correction. Rescue rate is formalized as

$g_d \in T$0

Mixed-effects logistic regression was used for correctness and rescue, with random intercepts for the message and fixed effects for domain, model, condition, and interactions. Several reported effects were statistically strong, all with $g_d \in T$1: Gaming versus Forum had $g_d \in T$2, Social Media versus Forum had $g_d \in T$3, the Aligned Definition condition had $g_d \in T$4, and zero-shot correctness strongly predicted prompted correctness with $g_d \in T$5 (Casanova et al., 30 May 2026).

Decision stickiness is most visible in the rescue analysis. Rescue among zero-shot errors decreased as zero-shot confidence increased; per standard-deviation increase in confidence, rescue odds fell with $g_d \in T$6. Rescue probability dropped to approximately $g_d \in T$7 for confidence greater than $g_d \in T$8. The reported rescue-versus-confidence curves had an inverted-U form, with the high-confidence tail described as the most stubborn. Supplementary analysis on Jigsaw Unintended Bias showed uniformly less than $g_d \in T$9 rescue across models, indicating that stickiness was not an artifact of label bias in standard Jigsaw (Casanova et al., 30 May 2026).

6. Misaligned definitions, practical use, and limitations

DSF is especially relevant where models are asked to follow explicit but potentially imperfect definitions. Under misaligned definitions, models in the study followed the instructed scope: narrow definitions led to under-prediction by $e_{m,d} \in T$0 to $e_{m,d} \in T$1, and broad definitions led to over-prediction by $e_{m,d} \in T$2 to $e_{m,d} \in T$3. However, average confidence remained similar to the aligned-definition condition, approximately $e_{m,d} \in T$4–$e_{m,d} \in T$5. Confidence-based detection of misalignment therefore failed (Casanova et al., 30 May 2026). This establishes an important boundary condition for LLM-as-a-judge systems: prompt compliance can coexist with unchanged confidence even when the operational definition has shifted.

The study’s worked example illustrates how DSF is intended to be used. For a gaming-toxicity definition that includes identity-based hate or harassment, threats, extremist content, direct insults or flaming, or other offensive text that harms reasonable enjoyment of the game, one hypothetical model produced a broad description covering insults, harassment, threats, hate targeting identity groups, and messages that disrupt fair play or enjoyment. Across six encoders, its similarity scores were $e_{m,d} \in T$6, yielding $e_{m,d} \in T$7. A second model narrowed toxicity mainly to hate speech about race or gender, with scores $e_{m,d} \in T$8 and $e_{m,d} \in T$9. The interpretation given is that the first model’s internal concept better matches the dataset’s broader scope and should therefore be expected to perform better in zero-shot annotation under aligned definitions (Casanova et al., 30 May 2026).

The practical guidance is correspondingly conservative. High DSF suggests better zero-shot performance and more coherent application of intended criteria; low DSF suggests lower accuracy and limited gains from prompting, implying a need for more extensive validation and potentially fine-tuning. At the same time, high DSF reflects alignment to the stated wording, not necessarily global optimality, and DSF does not measure capability or reasoning quality in isolation (Casanova et al., 30 May 2026).

The reported limitations are methodological and interpretive. DSF depends on sentence embeddings and concept elicitation, and may be sensitive to prompt wording, encoder selection, and domain-specific phrasing. The evidence is correlational rather than causal; the paper identifies fine-tuning interventions that manipulate DSF while holding other factors constant as future work. It also notes that steerability limits may reflect deliberate design choices from instruction tuning or RLHF rather than pure capability constraints, and it leaves stronger misalignment, multi-turn correction strategies, learned similarity metrics, and broader task families such as multi-class labeling, span labeling, and open-ended judging for future study (Casanova et al., 30 May 2026).