MSME-GEO-Bench: Multi-Engine GEO Evaluation

• MSME-GEO-Bench is a multi-scenario, multi-engine benchmark designed for rigorous generative engine optimization evaluation using causal attribution.
• It employs a Twin Branch Evaluation Protocol that controls retrieval context to jointly measure semantic visibility and citation fidelity.
• The benchmark integrates engine-specific annotations and structured pipelines to ensure reproducible, deployment-oriented performance analyses.

Searching arXiv for the benchmark and its related papers to ground the article in the cited literature. MSME-GEO-Bench is a multi-scenario, multi-engine benchmark for Generative Engine Optimization (GEO) introduced alongside the MAGEO framework to support rigorous, deployment-oriented evaluation in realistic settings (Wu et al., 21 Apr 2026). It is grounded in real-world queries, provides $$(\text{Query}, \text{Engine}, \text{Source}, \text{Response})$$ quadruples, and is paired with the Twin Branch Evaluation Protocol and the DSV-CF metric suite to isolate the causal effect of document edits under a frozen retrieval list while jointly measuring semantic visibility and attribution accuracy. Within the GEO literature, it occupies a distinct position: unlike prior GEO evaluations centered on single-engine or per-instance measurement, it emphasizes causal attribution, unified visibility-and-fidelity measurement, and cross-engine variation in black-box generative engines (Wu et al., 21 Apr 2026).

1. Definition and motivation

MSME-GEO-Bench was created to address three gaps in prior GEO evaluation. The first is causal attribution: existing offline or semi-simulated pipelines often conflate retrieval drift with content edits. The second is unified measurement of influence and trustworthiness: surface visibility and semantic influence are commonly measured independently. The third is real-world, multi-scenario coverage: GEO impact depends on query intent, domain, and engine preferences. Accordingly, the benchmark is grounded in Everyday Life Information Seeking (ELIS) theory, spans decision-oriented queries across daily-life domains, and supports cross-engine evaluation with proprietary and open-weight engines (Wu et al., 21 Apr 2026).

The benchmark provides quadruples consisting of a query, an engine, a source document $d_{src}$, and a generated response with citations. It locks a real source document per query through closed-loop retrieval validation using Top-10 re-retrieval, so edits to that source can be linked to measurable changes in real engine answers. This design suggests that MSME-GEO-Bench is not merely a collection of prompts and outputs, but a controlled evaluation substrate for attribution-sensitive GEO research (Wu et al., 21 Apr 2026).

A central implication of the benchmark’s design is that GEO is treated as more than rank or citation maximization. The benchmark explicitly evaluates both whether a source becomes more visible in a generated answer and whether the content attributed to that source remains supported and faithful. In this sense, the benchmark operationalizes GEO as a joint visibility-and-fidelity problem rather than a pure exposure problem (Wu et al., 21 Apr 2026).

2. Composition and annotation structure

MSME-GEO-Bench is organized by a Hierarchical Life Domain taxonomy (HLD-QT) and covers 5 major domains and 15 sub-category query types, including Health and Well-being, Finance and Economy, Education and Growth, Life and Consumption, and Law and Civic Affairs (Wu et al., 21 Apr 2026). Queries are further labeled by interaction intent, such as guidance, complex reasoning, and fact-checking, and by query complexity. The paper mentions a test split used for manual checks, but it does not specify exact train/dev/test sizes or counts, and the overall dataset size is not explicitly reported.

Its data construction pipeline is deliberately constrained. Seed queries span the HLD-QT space; Tavily Search API retrieves Top-10 candidate documents; one document is randomly locked as the source; Gemini-3 Pro reverse-generates queries that the locked source can answer; and closed-loop retrieval validation keeps only samples for which the locked source reappears in the Top-10 upon re-retrieval. Gemini-3 Pro then annotates life-domain, intent, and complexity labels, after which structured prompting, rule-based filtering, and sampled human checks are applied. Manual inspection of the test split shows over 95% tag precision (Wu et al., 21 Apr 2026).

Three engines are explicitly covered for evaluation: GPT-5.2 (OpenAI), Gemini-3 Pro (Google), and Qwen-3 Max (open weights). Because engine tags are part of each quadruple, the benchmark supports engine-specific analysis rather than only pooled evaluation. This suggests a benchmark philosophy in which differences in citation style, rhetorical preference, and formatting preference are first-class experimental variables rather than nuisance factors (Wu et al., 21 Apr 2026).

3. Twin Branch Evaluation Protocol

The Twin Branch Evaluation Protocol is the benchmark’s principal causal mechanism. Its goal is to enable causal attribution of content edits in black-box generative engines by controlling retrieval context (Wu et al., 21 Apr 2026). The setup begins from a user query $q$ and a fixed retrieval list $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$ from a search engine. In the baseline branch, the engine uses $\mathcal{L}_{ret}$ as-is to generate $r_{base}$. In the optimization branch, a target document $d_{target}$ is uniformly sampled from $\mathcal{L}_{ret}$, semantically edited into $d^{*}$, and reinserted in place, yielding $$\mathcal{L}_{new} = \mathcal{L}_{ret}[d_{target} \leftarrow d^{*}]$$, from which the engine generates $d_{src}$0 (Wu et al., 21 Apr 2026).

The optimization objective is given as

$d_{src}$1

with the stated purpose of attributing any observed changes in visibility or fidelity to the edit itself rather than to retrieval drift. The protocol therefore freezes the retrieval list across branches except for the targeted in-situ replacement of the edited source (Wu et al., 21 Apr 2026).

The operational steps are explicit. For each quadruple, the evaluator obtains $d_{src}$2, $d_{src}$3, and the locked source $d_{src}$4; generates $d_{src}$5 from the original list; edits $d_{src}$6 into $d_{src}$7; substitutes it into the list and generates $d_{src}$8; computes DSV-CF and all sub-metrics on both branches; and attributes the deltas to the edit. The paper also notes that one may optionally iterate edits under a fidelity gate, including in MAGEO or other optimizers. A plausible implication is that the protocol can serve as a common harness for comparing heuristic and learned GEO systems without changing the causal assumptions of the evaluation (Wu et al., 21 Apr 2026).

4. DSV-CF and sub-metric system

MSME-GEO-Bench evaluates two complementary task axes: semantic visibility and citation fidelity / attribution accuracy (Wu et al., 21 Apr 2026). The unified objective is DSV-CF:

$d_{src}$9

where $q$0 is the visibility-quality tradeoff weight with default $q$1, $q$2 is the miscitation penalty severity with default $q$3, $q$4 and $q$5 are normalized aggregates of the visibility and semantic-impact components, and $q$6 is Attribution Accuracy. The paper states that the normalization specifics for $q$7 and $q$8 are not fully specified (Wu et al., 21 Apr 2026).

The Surface Semantic Visibility (SSV) side includes Word-Level Visibility (WLV), Decayed Positional Authority (DPA), Citation Prominence (CP), and Subjective Impression (SI). WLV is defined as

$q$9

DPA is defined as

$\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$0

The paper notes that the original DPA equation appears to have a missing brace. Conceptually, earlier sentences receive larger weight through exponential decay by sentence position. CP is an LLM-judged visibility measure for citations in headings, bullets, bold, or other structured regions, while SI is an LLM-estimated measure of the target source’s perceived importance (Wu et al., 21 Apr 2026).

The Intrinsic Semantic Impact (ISI) side includes Attribution Accuracy (AA), Response-level Faithfulness ($\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$1), Key-Point Coverage (KC), and Answer Dominance (AD). AA checks whether claims attributed to the target source are supported by that source. $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$2 checks whether the optimized document remains semantically faithful to the original. KC measures recall of key points from the target document in the engine response. AD assesses whether the target source is presented as the primary solution, especially in comparative or recommendation queries (Wu et al., 21 Apr 2026).

The paper further reports that LLM-as-judge aligns with human evaluation, with Spearman $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$3 for DSV-CF, $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$4 for WLV, and $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$5 for CF, with $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$6. This supports the use of automatic judging for benchmark-scale evaluation, although it does not eliminate model-specific scoring bias (Wu et al., 21 Apr 2026).

5. Systems evaluated and empirical behavior

MSME-GEO-Bench is used to compare MAGEO against heuristic baselines from the GEO repository, including Authoritative, Citing Credible Sources, Statistics Addition (Stats Optimization), Quotation Addition (More Quotes), Easy-to-Read (Simple Language), Fluent, Unique Words, Technical Terms, and Keyword Optimization (SEO Optimize) (Wu et al., 21 Apr 2026). The benchmark is thus not restricted to a single optimization paradigm; it can evaluate both heuristic rewriting strategies and a multi-agent, strategy-learning system.

On GPT-5.2, the benchmark reports that the no-GEO baseline has WLV 1.00 and DPA 1.33; the best heuristics include More Quotes with WLV 1.33 and DPA 1.37, and Authoritative with WLV 1.29 and DPA 1.29; MAGEO (Main, Ours) reaches WLV 4.52, DPA 4.52, CP 6.93, SI 7.82, AA 7.96, FA 8.17, KC 7.85, and AD 7.54. The corresponding ablations are lower: without Engine Preference, WLV 2.08 and DPA 2.10; without Skill Bank, WLV 1.41 and DPA 1.57 (Wu et al., 21 Apr 2026).

On Gemini-3 Pro, the benchmark reports None at WLV 1.00 and DPA 1.00; the best heuristics include Technical Terms at WLV 1.29 and DPA 1.29, and Stats Optimization at WLV 1.25 and DPA 1.25; MAGEO (Main) reaches WLV 5.30, DPA 5.30, CP 7.44, SI 8.17, AA 8.03, FA 7.93, KC 7.54, and AD 7.11. On Qwen-3 Max, None is WLV 1.00 and DPA 1.00; More Quotes is the best heuristic with WLV 1.33 and DPA 1.16; MAGEO (Main) reaches WLV 3.84, DPA 3.84, CP 5.89, SI 6.65, AA 6.77, FA 6.94, KC 6.67, and AD 6.41 (Wu et al., 21 Apr 2026).

The ablation study attributes large portions of MAGEO’s gains to engine-specific preference modeling and to the Skill Bank. Removing engine-specific preference modeling substantially reduces performance, with an example drop of around $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$7 on GPT-5.2, while removing the Skill Bank yields a drop of about $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$8 on GPT-5.2. The paper also reports that gains peak around Version 5, described as “over-optimization fatigue,” motivating early stopping. Composite heuristics improve over single heuristics but remain far below MAGEO; for example, on GPT-5.2, stacked heuristics reach WLV 1.90 versus MAGEO 4.52 (Wu et al., 21 Apr 2026).

A representative case study uses the query “How can we mitigate the impact of ocean acidification on coral reef ecosystems?” for Gemini-3 Pro and reports post-optimization deltas of SI $\mathcal{L}_{ret} = \{d_1, \ldots, d_K\}$9, WLV $\mathcal{L}_{ret}$0, DPA $\mathcal{L}_{ret}$1, CP $\mathcal{L}_{ret}$2, AA $\mathcal{L}_{ret}$3, FA $\mathcal{L}_{ret}$4, KC $\mathcal{L}_{ret}$5, and AD $\mathcal{L}_{ret}$6 (Wu et al., 21 Apr 2026).

6. Engine-specific preference modeling, reproducibility, and limitations

MSME-GEO-Bench is designed for engine-specific evaluation because the quadruples explicitly bind queries, sources, responses, and engine tags. Within the MAGEO framework, a Preference Agent analyzes large-scale quadruples to construct an engine-specific profile $\mathcal{L}_{ret}$7 capturing tendencies such as statistical density, formatting, and rhetoric (Wu et al., 21 Apr 2026). The paper characterizes Gemini-3 Pro as favoring compact, structured evidence such as tables, URLs, and headings; GPT-5.2 as authority-seeking with heavy use of tables, headings, and reference-like signals, but with higher hallucination risk in citations; and Qwen-3 Max as didactic and safety-aware, with layered prose, bullet lists, and fewer fabricated references.

For usage, the benchmark requires the quadruples and labels, Tavily Search API for retrieval and closed-loop validation, target generative engines, and an LLM-as-judge for DSV-CF sub-metrics. The paper states that the repository provides the multi-agent optimization loop and evaluation pipeline, but specific script names and CLI details are not given in the text. It also states that MSME-GEO-Bench is released, while the specific dataset hosting URL and license are not explicitly provided (Wu et al., 21 Apr 2026).

Several limitations are explicit. Current size and category distribution limit fine-grained subgroup analyses, and exact dataset counts are not reported. Gemini-3 Pro is used for reverse query generation and annotation, so model-specific bias may remain despite safeguards. Engine behaviors drift over time, and the current benchmark is text-only rather than multimodal. There is also no formal analysis of skill generalization across unseen scenarios and no learning curve quantifying efficiency improvements versus accumulated experience (Wu et al., 21 Apr 2026).

The benchmark’s identity should also be distinguished from similarly named resources. “GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization” defines a benchmark for ranking-manipulation attacks and explicitly states that it does not use the term “MSME-GEO-Bench” (Nimase et al., 27 May 2026). The earlier “GEO: Generative Engine Optimization” paper introduces GEO-bench but likewise does not name or define MSME-GEO-Bench (Aggarwal et al., 2023). This contrast indicates that MSME-GEO-Bench is a later benchmark line focused on multi-engine, causal, fidelity-aware GEO evaluation rather than on single-engine visibility benchmarking or ranking-manipulation attacks.