---
title: MAGIC Benchmark for Conflict Localization
url: https://www.emergentmind.com/topics/magic-rich-benchmark
type: topic
---

# MAGIC Benchmark for Conflict Localization

The designation **“Magic-RICH Benchmark”** is not a benchmark name used in the relevant paper. In the benchmark literature summarized here, the relevant resource is **MAGIC**, “A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts,” which is designed to test retrieval-augmented generation systems and LLMs on detecting and localizing contradictions between two textual contexts that have been generated from knowledge-graph structures [2507.21544]. The likely source of confusion is **RICH** (Chen et al., 2022), a prior QA-centric study about knowledge conflicts, whereas MAGIC is graph-based, multi-hop, question-free, and targets inter-context conflicts and conflict localization. Separately, the string **“Magic-RICH”** also appears in an unrelated nuclear-structure context as “magic numbers from tensor blocking in neutron-rich nuclei,” which is not a benchmark and belongs to nuclear theory rather than RAG evaluation [1907.07902].

## 1. Terminological status and conceptual scope

MAGIC targets **inter-context conflict**: given two contexts, determine whether they contradict and, if so, where [2507.21544]. The paper distinguishes three conflict loci in RAG systems: conflicts between retrieved documents, conflicts between context and model memory, and conflicts within memory. MAGIC focuses specifically on the first of these. Robust systems, in this formulation, should detect if two sources disagree and pinpoint conflicting facts.

The paper explicitly contrasts MAGIC with RICH. RICH is described as a QA-centered calibration study that simulates conflicts by manipulating retrieved answers or evidence and examines model confidence. MAGIC is distinct in four stated respects: **graph-based construction**, **multi-hop emphasis**, **question-free context pairs**, and **fine-grained evaluation by hop-length and number of conflicts**, together with **explicit localization scoring** [2507.21544]. This distinction is central to understanding why “Magic-RICH Benchmark” is best treated as a naming confusion rather than as the title of a single benchmark.

A second source of ambiguity arises from the unrelated nuclear-physics usage of “Magic-RICH.” In that domain, “Magic-RICH” abbreviates **“magic numbers from tensor blocking in neutron-rich nuclei”**, a framework for explaining the newly observed magic numbers \(N=6,14,16,32,34\) through tensor blocking effects [1907.07902]. That usage has no connection to LLM benchmarking, RAG, or inter-context contradiction detection.

## 2. Formalization of MAGIC

MAGIC is built on a knowledge-graph formulation. A knowledge graph is defined as

$$
G = (V, R, E),
$$

where \(V\) is a set of entities, \(R\) is a set of relation types, and \(E \subseteq V \times R \times V\) is a set of directed, typed edges. A triple \(e \in E\) is \(e = (s, r, o)\) with subject \(s \in V\), relation \(r \in R\), and object \(o \in V\) [2507.21544].

Multi-hop reasoning is defined through paths. A path of length \(k\) is

$$
p = (v_0, r_1, v_1, \dots, r_k, v_k)
$$

where \((v_{i-1}, r_i, v_i) \in E\) for \(i = 1,\dots,k\). Single-hop corresponds to \(k=1\); multi-hop corresponds to \(k \ge 2\). MAGIC constructs two contexts \(C_1\) and \(C_2\) from two subgraphs \(G_1 = (V_1, R_1, E_1)\) and \(G_2 = (V_2, R_2, E_2)\), which are extracted from \(G\) and then verbalized into passages \(T(G_1)\) and \(T(G_2)\). Facts in a context are the triples of its subgraph, denoted \(F(C_i)=E_i\) [2507.21544].

The benchmark defines an **inter-context conflict** when there exist \(f_1 \in F(C_1)\) and \(f_2 \in F(C_2)\) such that \(f_1\) entails the negation of \(f_2\), or the pair violates a consistency constraint \(\psi\). Formally,

$$
\exists f_1, f_2: f_1 \vDash \neg f_2,
$$

or

$$
(f_1, f_2) \nvDash \psi.
$$

In multi-hop cases, the conflicting facts may be entailed by paths \(\pi_1\) and \(\pi_2\) of length at least 2, written as \(f_1 \in Cl_k(G_1)\) and \(f_2 \in Cl_k(G_2)\) for some closure operator \(Cl_k\) over paths up to length \(k \ge 2\) [2507.21544].

The framework also informally aims for contexts that are “similar yet distinct.” The paper gives a notional similarity measure between edge sets via the Jaccard index,

$$
S(G_1, G_2) = \frac{|E_1 \cap E_2|}{|E_1 \cup E_2|},
$$

while also stating that no fixed threshold is prescribed [2507.21544]. This suggests that subtlety is intended to arise from substantial local overlap combined with carefully induced inconsistency rather than from grossly dissimilar passages.

## 3. Data generation pipeline

The paper describes a three-stage pipeline: **subgraph extraction from Wikidata5M**, **knowledge conflict generation via LLM-guided perturbations plus human-in-the-loop quality control**, and **KG-to-text conversion with automatic verification** [2507.21544].

In the first stage, the source KG is **Wikidata5M**, containing approximately **20M triples**. Preprocessing removes poorly defined entities, approximately **4k**, the top **30** extremely high-degree “hub” concepts, and non-informative nodes. Seed triples are selected from a curated set of **46 relations** out of **825**, grouped into seven domains: **Human, Geography, Organization, Creative Work, Class/Concept, Cause-Effect, General**. Subgraphs are extracted with **depth-first search** from the seed’s subject. Structural constraints include \(|E_{\text{subgraph}}| \le 15\), node out-degree and in-degree within the subgraph of at most **5**, and randomly varied DFS depth to diversify structures [2507.21544].

In the second stage, the goal is to transform an extracted subgraph \(G_s\) into a perturbed subgraph \(\tilde{G}_s\) such that the verbalizations \(T(G_s)\) and \(T(\tilde{G}_s)\) contain one or more conflicts. Perturbations may alter nodes, edges, or small chains. The LLM engine used for this step is **OpenAI o3-mini**, prompted with the seed triple and its surrounding subgraph as triples. The paper states that including the local neighborhood helps generate contextually coherent contradictions, whereas prompting without subgraph context leads to trivial or off-topic edits. Few-shot prompting is performed at the subgraph level, with **three validated demonstrations per relation**, specifically to discourage simple entity swaps and encourage multi-hop chains. For \(N\)-conflict instances, multiple non-overlapping perturbations are applied until the desired number of independent conflicts has been introduced. Human-in-the-loop quality control operates at two stages: curation of few-shot exemplars and filtering of generated outputs to discard incoherent, trivial, or overlapping conflicts and to ensure multi-hop chains for multi-hop cases [2507.21544].

In the third stage, **GPT-4o-mini (2024-07-18)** generates coherent paragraphs that verbalize all triples in each subgraph under a strict “include all and only the given facts” prompt. **Claude 3.7 Sonnet** is then used for automated verification; only “No error” outputs are kept. A human spot-check on **167 sampled instances** reported **conflict-triplet coverage \(\approx 95.21\%\)** and **subgraph-triplet coverage \(\approx 82.04\%\)**, which the paper interprets as indicating high fidelity of KG-to-text conversion [2507.21544].

## 4. Conflict taxonomy, benchmark composition, and task design

MAGIC factorizes complexity along two axes: **hop-length** and **number of conflicts across the two contexts**. Hop-length is divided into **Single-Hop** and **Multi-Hop**; the number of conflicts is divided into **1** versus **N**, where \(N\) can be **2, 3, or 4**. This yields four settings: **1-Single-Hop, N-Single-Hop, 1-Multi-Hop, N-Multi-Hop** [2507.21544].

| Setting | Counts |
|---|---:|
| Single-Hop: \(N=1\) | 208 |
| Single-Hop: \(N=2\) | 154 |
| Single-Hop: \(N=3\) | 80 |
| Single-Hop: \(N=4\) | 50 |
| Multi-Hop: \(N=1\) | 300 |
| Multi-Hop: \(N=2\) | 158 |
| Multi-Hop: \(N=3\) | 80 |
| Multi-Hop: \(N=4\) | 50 |

The total dataset size is **1,080 examples**, in **English**, built from **Wikidata5M**, with coverage across the seven semantic domains and 46 selected relations [2507.21544]. The paper states that many contexts are relatively long because of multi-hop and description-rich generation, and it evaluates the whole set rather than reporting specific train/dev/test splits.

The benchmark supports three tasks: **conflict detection (Identification)**, **conflict localization (Pinpointing)**, and **Explanation**. The scoring focuses on detection and localization. The prompting strategy is explicitly multi-step: first ask whether a conflict is present, then ask for conflict count and reason, and finally request exact Sentence A / Sentence B pairs. The paper reports that this multi-step prompt outperforms a binary yes/no prompt by **up to 39.41% ID improvements on tested models** [2507.21544].

The evaluation metrics are strict. For identification, three independent inference runs are used. If \(y_{ir} \in \{0,1\}\) indicates correct conflict detection for instance \(i\) in run \(r \in \{1,2,3\}\), then

$$
ID_i = 1 \text{ if } \max_r y_{ir} = 1, \text{ else } 0,
$$

and the reported score is

$$
ID = \frac{1}{N}\sum_{i=1}^{N} ID_i.
$$

For localization, the score is **1** only if the model exactly identifies all conflicting sentence locations across the pair; otherwise it is **0**, and the average is reported over instances. In multi-conflict cases, partial localization is scored **0** [2507.21544]. The paper notes that precision, recall, and \(F_1\) are not reported.

Illustrative examples clarify the taxonomy. A **1-Single-Hop entity-level contradiction** is given by two contexts assigning different captains to the same tour. A **1-Single-Hop relation-level contradiction** uses **separated_from** versus **merged_with** for the same university pairs. A **1-Multi-Hop** example uses a chain of **equivalent_to** relations in one context and **different_from** in the other so that a contradiction only appears after chaining relations across at least two hops [2507.21544].

## 5. Empirical results and observed failure modes

The models evaluated without task-specific training are **Mixtral-8x7B Instruct**, **Llama 3.1 70B Instruct**, **GPT-4o-mini**, **o1**, and **Claude 3.5 Haiku**; the appendix additionally includes **Mistral 7B** and **Llama 3.1 8B** [2507.21544]. Comparison datasets are **MAGIC**, **ECON**, and **WikiContradict**.

A central reported finding is that MAGIC is harder than both ECON and WikiContradict in aggregate. Average **Identification** scores are **74.73** on ECON, **69.93** on WikiContradict, and **59.07** on MAGIC. Average **Localization** scores are **57.09**, **55.74**, and **36.42**, respectively [2507.21544]. For MAGIC specifically, the reported identification scores are **37.92** for Mixtral, **73.83** for Llama 3.1 70B, **31.94** for Claude 3.5 Haiku, **83.61** for GPT-4o-mini, and **68.06** for o1. The corresponding localization scores are **17.40**, **37.89**, **22.04**, **55.00**, and **49.72** [2507.21544].

The paper further states that **single-hop is markedly easier** for both identification and localization, while **multi-hop sharply reduces localization** because contradictions are distributed across chains and multiple sentences. Increasing the number of conflicts generally **increases identification** because the contradiction becomes more salient, but **decreases localization** because all spans must be pinpointed exactly [2507.21544].

Model-specific behaviors are described in detail. **GPT-4o-mini** is the strongest overall on MAGIC with **ID 83.61** and **LOC 55.00**. **o1** is conservative on ambiguous, multi-hop cases and sometimes predicts “no conflict,” missing subtle contradictions. **Claude 3.5 Haiku** drops substantially on MAGIC, indicating difficulty with multi-hop chains. **Llama 3.1 70B** tends to produce very long outputs and may over-detect, harming localization precision. **Mixtral’s** low localization partly stems from poor initial detection [2507.21544].

Additional analyses refine this picture. At the relation level, **captain** and **mother** conflicts are relatively easy, whereas **work location** and **father** are harder across models. At the domain level, **Class/Concept** relations such as **subclass_of** and **different_from** are easier, while **Organization** is uneven across models. A quantile analysis shows that both identification and especially localization degrade as total context length increases. Pairs spanning multiple domains make detection easier but localization harder. The paper also reports a **parametric knowledge effect**: when items are split by whether relevant triples are “known” to a model through separate verification questions, **GPT-4o-mini** and **Llama 3.1** show higher identification on “known” items, suggesting reliance on parametric knowledge even in an inter-context conflict task [2507.21544].

## 6. Relation to prior work, limitations, and the unrelated nuclear-physics usage

Relative to prior work, MAGIC is positioned as a complement to and extension of **RICH**, **ECON**, and **WikiContradict**. Against QA-only and entity-substitution datasets, the paper argues that MAGIC’s **question-free** and **graph-grounded** construction supports subtler contradictions, explicit analysis by hop-length and number of conflicts, and strict localization evaluation [2507.21544]. Against ECON, the paper emphasizes scale and difficulty: ECON has **168 instances**, many **1-Single-Hop** conflicts, whereas MAGIC has **1,080** instances and is more multi-hop heavy. Against WikiContradict, MAGIC’s multi-hop subset is reported to be harder in aggregate on both identification and localization [2507.21544].

The paper also states several limitations. The benchmark is built on **Wikidata5M** alone; extending to **DBpedia**, **YAGO**, or domain-specific KGs could broaden coverage and reduce bias. Real-world corpora may contain conflict types such as **temporal revision histories** and **probabilistic claims** that are not explicitly modeled. There is a **synthetic-to-natural gap** because the contexts are LLM-generated from KGs, even though human and automatic checks and the reported **naturalness \(\sim 4.08\)** and **realism \(\sim 4.26\)** scores indicate substantial fidelity. Localization evaluation currently requires **strict human scoring**, and the parametric-knowledge analysis is described as **coarse** [2507.21544].

The expression “Magic-RICH” therefore requires explicit disambiguation. In the benchmark domain, the appropriate referent is **MAGIC**, not “Magic-RICH.” In nuclear theory, **“Magic-RICH: magic numbers from tensor blocking in neutron-rich nuclei”** denotes a framework in which tensor blocking enlarges shell gaps and helps explain the newly observed magic numbers \(N=6,14,16,32,34\) [1907.07902]. The overlap in wording is purely nominal. A plausible implication is that searches for “Magic-RICH Benchmark” conflate three distinct items: the MAGIC benchmark for inter-context conflicts, the earlier RICH QA-centric conflict study, and the unrelated nuclear-physics phrase “Magic-RICH.”

Source: https://www.emergentmind.com/topics/magic-rich-benchmark