MAGIC Benchmark for Conflict Localization

• The paper introduces MAGIC as a graph-based, multi-hop benchmark that targets inter-context conflicts by pinpointing contradictory facts between knowledge-graph derived passages.
• The methodology employs a three-stage pipeline: subgraph extraction from Wikidata5M, LLM-guided perturbations with human quality control, and strict KG-to-text conversion.
• Empirical analysis reveals that MAGIC is more challenging than related datasets, with multi-hop and N-conflict settings exposing key limitations in current LLMs and RAG systems.

The designation “Magic-RICH Benchmark” is not a benchmark name used in the relevant paper. In the benchmark literature summarized here, the relevant resource is MAGIC, “A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts,” which is designed to test retrieval-augmented generation systems and LLMs on detecting and localizing contradictions between two textual contexts that have been generated from knowledge-graph structures (Lee et al., 29 Jul 2025). The likely source of confusion is RICH (Chen et al., 2022), a prior QA-centric study about knowledge conflicts, whereas MAGIC is graph-based, multi-hop, question-free, and targets inter-context conflicts and conflict localization. Separately, the string “Magic-RICH” also appears in an unrelated nuclear-structure context as “magic numbers from tensor blocking in neutron-rich nuclei,” which is not a benchmark and belongs to nuclear theory rather than RAG evaluation (Tanihata et al., 2019).

1. Terminological status and conceptual scope

MAGIC targets inter-context conflict: given two contexts, determine whether they contradict and, if so, where (Lee et al., 29 Jul 2025). The paper distinguishes three conflict loci in RAG systems: conflicts between retrieved documents, conflicts between context and model memory, and conflicts within memory. MAGIC focuses specifically on the first of these. Robust systems, in this formulation, should detect if two sources disagree and pinpoint conflicting facts.

The paper explicitly contrasts MAGIC with RICH. RICH is described as a QA-centered calibration study that simulates conflicts by manipulating retrieved answers or evidence and examines model confidence. MAGIC is distinct in four stated respects: graph-based construction, multi-hop emphasis, question-free context pairs, and fine-grained evaluation by hop-length and number of conflicts, together with explicit localization scoring (Lee et al., 29 Jul 2025). This distinction is central to understanding why “Magic-RICH Benchmark” is best treated as a naming confusion rather than as the title of a single benchmark.

A second source of ambiguity arises from the unrelated nuclear-physics usage of “Magic-RICH.” In that domain, “Magic-RICH” abbreviates “magic numbers from tensor blocking in neutron-rich nuclei”, a framework for explaining the newly observed magic numbers $N=6,14,16,32,34$ through tensor blocking effects (Tanihata et al., 2019). That usage has no connection to LLM benchmarking, RAG, or inter-context contradiction detection.

2. Formalization of MAGIC

MAGIC is built on a knowledge-graph formulation. A knowledge graph is defined as

$G = (V, R, E),$

where $V$ is a set of entities, $R$ is a set of relation types, and $E \subseteq V \times R \times V$ is a set of directed, typed edges. A triple $e \in E$ is $e = (s, r, o)$ with subject $s \in V$, relation $r \in R$, and object $o \in V$ (Lee et al., 29 Jul 2025).

Multi-hop reasoning is defined through paths. A path of length $G = (V, R, E),$0 is

$G = (V, R, E),$1

where $G = (V, R, E),$2 for $G = (V, R, E),$3. Single-hop corresponds to $G = (V, R, E),$4; multi-hop corresponds to $G = (V, R, E),$5. MAGIC constructs two contexts $G = (V, R, E),$6 and $G = (V, R, E),$7 from two subgraphs $G = (V, R, E),$8 and $G = (V, R, E),$9, which are extracted from $V$0 and then verbalized into passages $V$1 and $V$2. Facts in a context are the triples of its subgraph, denoted $V$3 (Lee et al., 29 Jul 2025).

The benchmark defines an inter-context conflict when there exist $V$4 and $V$5 such that $V$6 entails the negation of $V$7, or the pair violates a consistency constraint $V$8. Formally,

$V$9

or

$R$0

In multi-hop cases, the conflicting facts may be entailed by paths $R$1 and $R$2 of length at least 2, written as $R$3 and $R$4 for some closure operator $R$5 over paths up to length $R$6 (Lee et al., 29 Jul 2025).

The framework also informally aims for contexts that are “similar yet distinct.” The paper gives a notional similarity measure between edge sets via the Jaccard index,

$R$7

while also stating that no fixed threshold is prescribed (Lee et al., 29 Jul 2025). This suggests that subtlety is intended to arise from substantial local overlap combined with carefully induced inconsistency rather than from grossly dissimilar passages.

3. Data generation pipeline

The paper describes a three-stage pipeline: subgraph extraction from Wikidata5M, knowledge conflict generation via LLM-guided perturbations plus human-in-the-loop quality control, and KG-to-text conversion with automatic verification (Lee et al., 29 Jul 2025).

In the first stage, the source KG is Wikidata5M, containing approximately 20M triples. Preprocessing removes poorly defined entities, approximately 4k, the top 30 extremely high-degree “hub” concepts, and non-informative nodes. Seed triples are selected from a curated set of 46 relations out of 825, grouped into seven domains: Human, Geography, Organization, Creative Work, Class/Concept, Cause-Effect, General. Subgraphs are extracted with depth-first search from the seed’s subject. Structural constraints include $R$8, node out-degree and in-degree within the subgraph of at most 5, and randomly varied DFS depth to diversify structures (Lee et al., 29 Jul 2025).

In the second stage, the goal is to transform an extracted subgraph $R$9 into a perturbed subgraph $E \subseteq V \times R \times V$0 such that the verbalizations $E \subseteq V \times R \times V$1 and $E \subseteq V \times R \times V$2 contain one or more conflicts. Perturbations may alter nodes, edges, or small chains. The LLM engine used for this step is OpenAI o3-mini, prompted with the seed triple and its surrounding subgraph as triples. The paper states that including the local neighborhood helps generate contextually coherent contradictions, whereas prompting without subgraph context leads to trivial or off-topic edits. Few-shot prompting is performed at the subgraph level, with three validated demonstrations per relation, specifically to discourage simple entity swaps and encourage multi-hop chains. For $E \subseteq V \times R \times V$3-conflict instances, multiple non-overlapping perturbations are applied until the desired number of independent conflicts has been introduced. Human-in-the-loop quality control operates at two stages: curation of few-shot exemplars and filtering of generated outputs to discard incoherent, trivial, or overlapping conflicts and to ensure multi-hop chains for multi-hop cases (Lee et al., 29 Jul 2025).

In the third stage, GPT-4o-mini (2024-07-18) generates coherent paragraphs that verbalize all triples in each subgraph under a strict “include all and only the given facts” prompt. Claude 3.7 Sonnet is then used for automated verification; only “No error” outputs are kept. A human spot-check on 167 sampled instances reported conflict-triplet coverage $E \subseteq V \times R \times V$4 and subgraph-triplet coverage $E \subseteq V \times R \times V$5, which the paper interprets as indicating high fidelity of KG-to-text conversion (Lee et al., 29 Jul 2025).

4. Conflict taxonomy, benchmark composition, and task design

MAGIC factorizes complexity along two axes: hop-length and number of conflicts across the two contexts. Hop-length is divided into Single-Hop and Multi-Hop; the number of conflicts is divided into 1 versus N, where $E \subseteq V \times R \times V$6 can be 2, 3, or 4. This yields four settings: 1-Single-Hop, N-Single-Hop, 1-Multi-Hop, N-Multi-Hop (Lee et al., 29 Jul 2025).

Setting	Counts
Single-Hop: $E \subseteq V \times R \times V$7	208
Single-Hop: $E \subseteq V \times R \times V$8	154
Single-Hop: $E \subseteq V \times R \times V$9	80
Single-Hop: $e \in E$0	50
Multi-Hop: $e \in E$1	300
Multi-Hop: $e \in E$2	158
Multi-Hop: $e \in E$3	80
Multi-Hop: $e \in E$4	50

The total dataset size is 1,080 examples, in English, built from Wikidata5M, with coverage across the seven semantic domains and 46 selected relations (Lee et al., 29 Jul 2025). The paper states that many contexts are relatively long because of multi-hop and description-rich generation, and it evaluates the whole set rather than reporting specific train/dev/test splits.

The benchmark supports three tasks: conflict detection (Identification), conflict localization (Pinpointing), and Explanation. The scoring focuses on detection and localization. The prompting strategy is explicitly multi-step: first ask whether a conflict is present, then ask for conflict count and reason, and finally request exact Sentence A / Sentence B pairs. The paper reports that this multi-step prompt outperforms a binary yes/no prompt by up to 39.41% ID improvements on tested models (Lee et al., 29 Jul 2025).

The evaluation metrics are strict. For identification, three independent inference runs are used. If $e \in E$5 indicates correct conflict detection for instance $e \in E$6 in run $e \in E$7, then

$e \in E$8

and the reported score is

$e \in E$9

For localization, the score is 1 only if the model exactly identifies all conflicting sentence locations across the pair; otherwise it is 0, and the average is reported over instances. In multi-conflict cases, partial localization is scored 0 (Lee et al., 29 Jul 2025). The paper notes that precision, recall, and $e = (s, r, o)$0 are not reported.

Illustrative examples clarify the taxonomy. A 1-Single-Hop entity-level contradiction is given by two contexts assigning different captains to the same tour. A 1-Single-Hop relation-level contradiction uses separated_from versus merged_with for the same university pairs. A 1-Multi-Hop example uses a chain of equivalent_to relations in one context and different_from in the other so that a contradiction only appears after chaining relations across at least two hops (Lee et al., 29 Jul 2025).

5. Empirical results and observed failure modes

The models evaluated without task-specific training are Mixtral-8x7B Instruct, Llama 3.1 70B Instruct, GPT-4o-mini, o1, and Claude 3.5 Haiku; the appendix additionally includes Mistral 7B and Llama 3.1 8B (Lee et al., 29 Jul 2025). Comparison datasets are MAGIC, ECON, and WikiContradict.

A central reported finding is that MAGIC is harder than both ECON and WikiContradict in aggregate. Average Identification scores are 74.73 on ECON, 69.93 on WikiContradict, and 59.07 on MAGIC. Average Localization scores are 57.09, 55.74, and 36.42, respectively (Lee et al., 29 Jul 2025). For MAGIC specifically, the reported identification scores are 37.92 for Mixtral, 73.83 for Llama 3.1 70B, 31.94 for Claude 3.5 Haiku, 83.61 for GPT-4o-mini, and 68.06 for o1. The corresponding localization scores are 17.40, 37.89, 22.04, 55.00, and 49.72 (Lee et al., 29 Jul 2025).

The paper further states that single-hop is markedly easier for both identification and localization, while multi-hop sharply reduces localization because contradictions are distributed across chains and multiple sentences. Increasing the number of conflicts generally increases identification because the contradiction becomes more salient, but decreases localization because all spans must be pinpointed exactly (Lee et al., 29 Jul 2025).

Model-specific behaviors are described in detail. GPT-4o-mini is the strongest overall on MAGIC with ID 83.61 and LOC 55.00. o1 is conservative on ambiguous, multi-hop cases and sometimes predicts “no conflict,” missing subtle contradictions. Claude 3.5 Haiku drops substantially on MAGIC, indicating difficulty with multi-hop chains. Llama 3.1 70B tends to produce very long outputs and may over-detect, harming localization precision. Mixtral’s low localization partly stems from poor initial detection (Lee et al., 29 Jul 2025).

Additional analyses refine this picture. At the relation level, captain and mother conflicts are relatively easy, whereas work location and father are harder across models. At the domain level, Class/Concept relations such as subclass_of and different_from are easier, while Organization is uneven across models. A quantile analysis shows that both identification and especially localization degrade as total context length increases. Pairs spanning multiple domains make detection easier but localization harder. The paper also reports a parametric knowledge effect: when items are split by whether relevant triples are “known” to a model through separate verification questions, GPT-4o-mini and Llama 3.1 show higher identification on “known” items, suggesting reliance on parametric knowledge even in an inter-context conflict task (Lee et al., 29 Jul 2025).

6. Relation to prior work, limitations, and the unrelated nuclear-physics usage

Relative to prior work, MAGIC is positioned as a complement to and extension of RICH, ECON, and WikiContradict. Against QA-only and entity-substitution datasets, the paper argues that MAGIC’s question-free and graph-grounded construction supports subtler contradictions, explicit analysis by hop-length and number of conflicts, and strict localization evaluation (Lee et al., 29 Jul 2025). Against ECON, the paper emphasizes scale and difficulty: ECON has 168 instances, many 1-Single-Hop conflicts, whereas MAGIC has 1,080 instances and is more multi-hop heavy. Against WikiContradict, MAGIC’s multi-hop subset is reported to be harder in aggregate on both identification and localization (Lee et al., 29 Jul 2025).

The paper also states several limitations. The benchmark is built on Wikidata5M alone; extending to DBpedia, YAGO, or domain-specific KGs could broaden coverage and reduce bias. Real-world corpora may contain conflict types such as temporal revision histories and probabilistic claims that are not explicitly modeled. There is a synthetic-to-natural gap because the contexts are LLM-generated from KGs, even though human and automatic checks and the reported naturalness $e = (s, r, o)$1 and realism $e = (s, r, o)$2 scores indicate substantial fidelity. Localization evaluation currently requires strict human scoring, and the parametric-knowledge analysis is described as coarse (Lee et al., 29 Jul 2025).

The expression “Magic-RICH” therefore requires explicit disambiguation. In the benchmark domain, the appropriate referent is MAGIC, not “Magic-RICH.” In nuclear theory, “Magic-RICH: magic numbers from tensor blocking in neutron-rich nuclei” denotes a framework in which tensor blocking enlarges shell gaps and helps explain the newly observed magic numbers $e = (s, r, o)$3 (Tanihata et al., 2019). The overlap in wording is purely nominal. A plausible implication is that searches for “Magic-RICH Benchmark” conflate three distinct items: the MAGIC benchmark for inter-context conflicts, the earlier RICH QA-centric conflict study, and the unrelated nuclear-physics phrase “Magic-RICH.”