CoRE-Eval: Semantic Relation Benchmark

• CoRE-Eval is a benchmark that evaluates semantic relation reasoning in LLMs, explicitly testing for both meaningful relations and the absence of any relation.
• It uses a 203-question multiple‐choice format, balanced between related and unrelated pairs, to expose systematic 'semantic collapse' in model predictions.
• The evaluation employs metrics such as accuracy, balanced accuracy, ECE, and SCR to highlight practical implications for safety and high-stakes decision-making.

CoRE-Eval, in the usage introduced by "CORE: Comprehensive Ontological Relation Evaluation for LLMs," denotes the evaluation subset of CORE, a benchmark for ontological and semantic relation reasoning in LLMs. CORE comprises a 225,000-question multiple-choice dataset spanning 74 disciplines, while CoRE-Eval is a 203-question general-domain benchmark derived from that corpus and reserved for evaluation. Its defining feature is that it treats unrelatedness as a first-class semantic category: alongside standard relation types such as synonymy, hyponymy, meronymy, and cause–effect, it explicitly tests whether a model can recognize when no meaningful semantic relation exists between two concepts (Dwivedi et al., 6 Feb 2026).

1. Definition and motivating problem

CoRE-Eval is designed to assess two coupled capabilities. The first is conventional semantic relation reasoning: identifying and applying meaningful semantic relations such as synonymy, antonymy, agent–instrument, or class–instance. The second, and more distinctive, capability is reasoning about unrelatedness: correctly identifying cases in which no meaningful semantic relation applies between two concepts. The benchmark is therefore not limited to asking whether a model can find a valid relation; it also asks whether the model can refrain from constructing a relation when none exists (Dwivedi et al., 6 Feb 2026).

The framework is motivated by a gap in prior evaluation practice. Existing benchmarks largely emphasize positive relation recognition, analogy solving, entailment, or balanced datasets with meaningful internal structure. They rarely test whether a model can reject the premise that a relation exists. CoRE-Eval is built around the claim that this omission is consequential, because high-stakes reasoning often depends on distinguishing genuine structure from noise. The paper situates this issue in clinical decision support, finance, legal reasoning, and scientific inference, where a model that imposes structure on unrelated concepts may produce plausible but invalid reasoning chains (Dwivedi et al., 6 Feb 2026).

A central term introduced in the work is semantic collapse. This denotes the systematic tendency of models to impose a relation where none exists. The paper explicitly distinguishes this failure from ordinary factual hallucination. The relevant failure mode is described as “confident construction of spurious relational structures rather than factual hallucination.” In this framing, the core safety issue is not invention of nonexistent entities or facts, but overconfident relational fabrication grounded in weak association, co-occurrence, or superficial analogy (Dwivedi et al., 6 Feb 2026).

2. Benchmark construction and semantic coverage

CORE is a 225,000-question multiple-choice dataset spanning 74 disciplines across STEM, humanities, and social sciences. It is intended for fine-tuning, instruction-tuning, and evaluation. CoRE-Eval is the evaluation subset: 203 general-domain multiple-choice items, created from the larger pool but kept separate for clean evaluation. The benchmark was initially authored as 250 questions, each with a correct answer and a human-written explanation, then filtered through a three-pass expert review process. The final 203 items are those for which annotators achieved perfect inter-annotator agreement, with Cohen’s $\kappa = 1.0$ (Dwivedi et al., 6 Feb 2026).

The benchmark is nearly balanced between related and unrelated questions: 103 related items and 100 unrelated items. It is also split into an open subset of 102 questions and a blind subset of 101 questions. This balance is methodologically important because it suppresses simple base-rate heuristics and permits separate performance estimates for related versus unrelated reasoning (Dwivedi et al., 6 Feb 2026).

Each question follows an analogy-style multiple-choice format. A reference pair $(A:B)$ is provided, together with a target pair $(C: ?)$ and four answer options. The task is to select the option that instantiates the same relation type as the reference. For unrelated questions, the construction is more stringent: the reference pair itself is unrelated, the correct option is the one that makes the target pair unrelated as well, and the distractors are chosen so that they could plausibly form some relation with the target term. This prevents trivial strategies such as selecting the oddest or least associated option, and forces explicit recognition of the absence of relation (Dwivedi et al., 6 Feb 2026).

The benchmark covers 24 semantic relation types. These are listed below exactly as specified in the paper.

Relation type	Relation type	Relation type
Agent–instrument	Antonymy (complementary)	Antonymy (converse)
Antonymy (gradable)	Cause–effect	Class–instance
Co-hyponymy	Entailment	Function–object
Homonymy	Hyponymy	Incompatibility
Material–object	Meronymy	Metonymy
Near-synonymy	Part–substance	Place–event
Polysemy	Presupposition	Synonymy
Troponymy	Whole–process–step	Unrelated pairs

The inclusion of “unrelated pairs” in this inventory is the key conceptual move. Rather than treating negative cases as residuals or malformed prompts, CoRE-Eval formalizes unrelatedness as a semantic outcome in its own right. This suggests an evaluation regime in which semantic competence includes not only taxonomic and relational knowledge, but also reliable rejection of unwarranted linkage (Dwivedi et al., 6 Feb 2026).

3. Evaluation protocol and metrics

The reported experiments evaluate 29 state-of-the-art LLMs from major developers, including models from OpenAI, Google, Anthropic, Meta, DeepSeek, Mistral, Amazon, ZAI, and others. Proprietary models were accessed via APIs using deterministic inference and recommended settings; open-source models were run locally on standardized hardware with default generation settings. All models received the same prompt template, which asks for a single JSON object containing "answer", "confidences_by_option", and several introspective fields such as rationale, time_to_think, difficulty, and hallucinating. The paper notes that these introspective fields are not treated as ground truth about internal states, but rather as signals for analyzing self-reported confidence and metacognition (Dwivedi et al., 6 Feb 2026).

The benchmark uses standard accuracy and balanced accuracy. With $Q$ the set of $N$ questions, $y_i$ the ground-truth label, and $\hat{y}_i$ the model prediction, correctness is defined by $\mathbf{1}[\hat{y}_i = y_i]$, and accuracy is

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i].$$

Balanced accuracy averages accuracy by relation type:

$$\text{Balanced Accuracy} = \frac{1}{|R|} \sum_{r \in R} \text{Accuracy}_r.$$

This is relevant because the benchmark spans 24 semantic relation types with differing frequencies in the broader corpus (Dwivedi et al., 6 Feb 2026).

Calibration is quantified using Expected Calibration Error (ECE), computed over 10 confidence bins:

$(A:B)$0

ECE is reported overall and separately for related and unrelated items. To capture high-confidence mistakes, the paper also defines Overconfident Error Rate (OER):

$(A:B)$1

Here $(A:B)$2 is the rescaled confidence assigned to the chosen answer (Dwivedi et al., 6 Feb 2026).

Semantic collapse is operationalized through Semantic Collapse Rate (SCR). The paper states that the intended quantity is the fraction of unrelated items that are misclassified as related, while also noting a slight mismatch between the narrative description and the printed notation. The printed expression is

$(A:B)$3

with the accompanying explanation making clear that the substantive target is the rate at which models force unrelated pairs into one of the relational categories. In effect, SCR is used to quantify systematic spurious relation construction on unrelated items (Dwivedi et al., 6 Feb 2026).

4. Human baseline and model performance

A human baseline was collected from more than 1,000 participants in India, ranging from undergraduate to postdoctoral education levels, under blind evaluation conditions. Human performance on the 203-item benchmark was high and especially strong on unrelated items, which the paper treats as evidence that unrelatedness recognition is not an intrinsically pathological task (Dwivedi et al., 6 Feb 2026).

Category	Accuracy	Balanced Accuracy	Mean Entropy
Overall	92.6%	90.1%	0.45
Related pairs	90.2%	89.9%	0.58
Unrelated pairs	95.1%	95.1%	0.31

By contrast, the 29 evaluated LLMs achieved 48.25% to 70.90% overall accuracy. The aggregate pattern is strongly asymmetric. On related pairs, models achieved 86.50% to 100% accuracy, which is near ceiling. On unrelated pairs, performance collapsed to 0% to 41.35%. Despite this asymmetry, mean confidence remained high and similar across conditions, roughly 92% to 95% overall, 93% to 95% on related pairs, and 91% to 94% on unrelated pairs (Dwivedi et al., 6 Feb 2026).

The same asymmetry appears in calibration. Overall ECE ranged from 24.4% to 51.1%. On related pairs it ranged from 8.0% to 15.0%, but on unrelated pairs it rose to 24.0% to 51.0%, a 2–4$(A:B)$4 increase. The reported Overconfidence Error Rate on unrelated items was 29.1% to 51.75%, indicating that a substantial fraction of errors were committed with confidence at or above 0.75. Mean SCR across models was approximately 37.6%, indicating systematic rather than random imposition of spurious relations (Dwivedi et al., 6 Feb 2026).

The benchmark also reports performance on the full 225K CORE multiple-choice dataset. There, model accuracy drops to approximately 2%. The paper interprets this as evidence that general-domain competence on the 203-item benchmark does not transfer to discipline-rich, domain-specific semantic reasoning at scale. Difficulty stratification shows an additional anomaly: reported model accuracy rises from easy to medium questions, but on hard questions collapses to 0%, whereas humans still remain substantially above chance. This suggests a sharp capability cliff rather than smooth degradation (Dwivedi et al., 6 Feb 2026).

5. Semantic collapse as a failure mode

The benchmark’s main analytical claim is that current LLMs do not merely err on unrelated items; they often generate internally coherent but invalid relational narratives. The paper illustrates this with a prompt in which “Hospital is to flying as wolf is to _?” contains no meaningful relation in the reference pair, yet models commonly select an answer and justify it via narratives about group membership or containment. The defect is therefore not syntactic task failure but an inability to represent “no relation” as a stable outcome (Dwivedi et al., 6 Feb 2026).

This failure pattern appears across model families, parameter scales, and training styles. The paper states that it is observed in models from OpenAI, Google, Anthropic, Meta, DeepSeek, Mistral, and others; across scales from approximately 8B to 405B parameters; and regardless of whether models are standard supervised, RLHF-based, or reasoning-tuned. A plausible implication is that the problem is not idiosyncratic to a single model family, but reflects broader architectural, objective-level, and evaluation-level biases (Dwivedi et al., 6 Feb 2026).

The paper proposes three such biases. First is an architectural bias: transformer-style sequence modeling and softmax decision mechanisms encourage selection of some coherent option rather than explicit relational rejection. Second is an objective bias: cross-entropy training favors confident single-answer prediction, not abstention or negation of an assumed relation. Third is an evaluation bias: multiple-choice settings compel choice among options even when the semantically correct outcome is relation absence. The paper also relates the phenomenon to sycophancy and to a decoupling between coherence and correctness, where internally consistent explanations are treated as if they licensed the existence of a real semantic relation (Dwivedi et al., 6 Feb 2026).

The safety implications follow directly from this analysis. Because semantic collapse is not obvious factual hallucination, it may be harder to detect in deployment. The explanations are well-formed, plausible, and often high-confidence. The paper therefore treats unrelatedness reasoning as a critical, under-evaluated frontier for LLM evaluation and safety, especially in domains where weak analogy or spurious association can mislead downstream human decision-makers (Dwivedi et al., 6 Feb 2026).

6. Limitations, extensions, and terminological ambiguity

The authors identify four explicit limitations. First, the benchmark is English-only. Second, evaluation is restricted to multiple-choice format, so open-ended generation behavior may differ. Third, the dataset is text-only and does not address multimodal unrelatedness. Fourth, while the full CORE corpus spans 74 disciplines, the 203-item CoRE-Eval benchmark is general-domain rather than strongly domain-specific. The paper notes that the approximately 2% accuracy on the full 225K dataset already hints at the difficulty of domain-specific semantic reasoning, but does not itself constitute a specialized benchmark design (Dwivedi et al., 6 Feb 2026).

Several future directions are outlined. Preliminary experiments reportedly suggest even larger performance gaps in non-English languages, motivating multilingual and low-resource extensions. Early fine-tuning results on the 225K CORE dataset reportedly show improvements in relational and general reasoning, with future work planned on whether such tuning reduces semantic collapse and improves calibration. The paper also proposes mechanistic studies using attention patterns and linear directions related to unanswerability, as well as architectural interventions such as no-relation tokens, dual pathways for relation inference versus relation rejection, and modified attention mechanisms that encode exclusion (Dwivedi et al., 6 Feb 2026).

The benchmark and associated resources are stated to be available at core.vaikhari.ai, with data and code hosted on Hugging Face and GitHub. The open-source release includes the 203-question benchmark, the larger 225K dataset, question texts, options, correct answers, and human-written explanations. The recommended evaluation procedure is to apply the standardized multiple-choice prompt from Appendix A and compute accuracy, balanced accuracy, ECE, OER, and SCR (Dwivedi et al., 6 Feb 2026).

The name CoRE-Eval is also used in unrelated contemporaneous work. In the LRM metacognition literature, it denotes a training-free, label-free self-evaluation mechanism based on Chain-of-Reasoning Embedding for early stopping in long chains of thought (Li et al., 8 Jul 2025). In multi-agent dialog research, CORE denotes the Conversational Robustness Evaluation Score, a task-agnostic metric for mode collapse, lexical repetition, and semantic stagnation under game-theoretic interaction (Pandey et al., 16 Aug 2025). The acronym also appears in full-path LLM-agent evaluation based on deterministic finite automata (Michelakis et al., 25 Sep 2025), and in code readability assessment as CoReEval (Ouédraogo et al., 18 Oct 2025). Within semantic-relation evaluation, however, CoRE-Eval refers specifically to the 203-item evaluation benchmark of CORE and to its emphasis on unrelatedness reasoning as an independent dimension of semantic competence (Dwivedi et al., 6 Feb 2026).