AmbiGraph-Eval: Benchmark Ambiguity in Graph Queries

• The paper introduces a taxonomy of graph-query ambiguities with six subcategories and presents 560 expert-verified examples.
• It proposes the AREA metric to reward models that generate multiple valid Cypher queries, ensuring full interpretation coverage.
• Empirical evaluations of nine LLMs reveal that even top models struggle to capture all plausible interpretations in ambiguous contexts.

Searching arXiv for AmbiGraph-Eval and closely related evaluation papers. AmbiGraph-Eval is a benchmark for evaluating whether LLMs can correctly handle ambiguity in natural-language graph queries, specifically in text-to-Cypher semantic parsing. It targets cases in which a user request is plausibly interpretable in multiple ways and where graph-structured data amplifies the ambiguity because meaning may reside in node attributes, edge types, relationship attributes, or graph neighborhoods rather than in a single flat schema location. The benchmark introduces a taxonomy of graph-query ambiguities, a curated dataset of ambiguous and unambiguous examples paired with expert-verified Cypher answers, and an execution-based metric designed to reward models that recover all valid interpretations rather than committing to only one plausible reading (Tian et al., 13 Aug 2025).

1. Problem formulation and scope

AmbiGraph-Eval studies ambiguity in text-to-Cypher / text-to-CQL graph query generation. The central task is not merely to translate natural language into an executable query, but to do so when the natural-language request admits multiple valid interpretations grounded in the same graph schema. In this setting, a model must recognize that the ambiguity is genuine, map each interpretation to the graph correctly, and generate Cypher queries whose execution results collectively cover the full target answer set (Tian et al., 13 Aug 2025).

The benchmark is motivated by the claim that ambiguity in graph semantic parsing is more difficult than in SQL-oriented settings because graph queries require not only schema grounding but also choice of traversal path, relationship semantics, comparison basis, aggregation scope, and syntactically correct Cypher realization. A phrase such as “best evaluated restaurant” can be ambiguous both at the level of semantic grounding and at the level of aggregation: the graph may contain multiple rating-like signals, and “best” may refer either to an individual maximum or to an aggregate across related nodes. This makes graph ambiguity simultaneously a problem of lexical grounding, relational interpretation, and query-program synthesis (Tian et al., 13 Aug 2025).

The paper further argues that modern LLM-based semantic parsers often exhibit self-preference bias in such cases: instead of signaling that multiple interpretations are valid, they tend to commit to one reading and emit a single precise Cypher query. A plausible implication is that the benchmark is intended as a stress test not only for semantic parsing accuracy but also for ambiguity detection and interpretation coverage.

2. Ambiguity taxonomy

AmbiGraph-Eval organizes graph-query ambiguity into three primary ambiguity types, each divided into Same-Entity and Cross-Entity cases, yielding six subcategories in total (Tian et al., 13 Aug 2025).

Primary type	Same-Entity	Cross-Entity
Attribute Ambiguity	Multiple plausible attributes for one entity	Unclear attribute basis for comparison across entities
Relationship Ambiguity	Multiple plausible relationships for one entity	Unclear relation-derived comparison criterion across entities
Attribute-Relationship Ambiguity	Unclear whether answer comes from an attribute or from relationships for one entity	Unclear whether comparison should use attributes or relational evidence across entities

Attribute Ambiguity occurs when a request refers to an entity property but multiple graph attributes could satisfy it. The representative example is asking for the citation count of a paper when the graph stores citation counts from multiple sources such as Google Scholar and Semantic Scholar. In the Same-Entity variant, the ambiguity is local to one entity. In the Cross-Entity variant, the ambiguity affects the basis of comparison, as in “Which paper has the highest citation count?” (Tian et al., 13 Aug 2025).

Relationship Ambiguity arises when the intended answer depends on the entity’s graph relationships, but the relevant relationship is underspecified. The paper’s examples include an architect’s “contributions”, which may correspond to different relation types, and “Who is the best doctor according to patient reviews?”, where the relevant review-derived signal is unclear. Same-Entity cases require choosing among multiple relations attached to one entity; Cross-Entity cases require choosing the correct comparison standard derived from relationships (Tian et al., 13 Aug 2025).

Attribute-Relationship Ambiguity is the most graph-specific category. Here, the ambiguity is whether the answer should be obtained from a node’s own property or from related entities or relationships. The canonical example is “What is the rating for the movie Inception?”, which may refer either to an intrinsic property such as IMDb rating or to review information reachable through graph traversal. The Cross-Entity version extends this to comparative questions such as “Which city hall has the highest evaluation?”, where both the signal source and the comparison procedure are ambiguous (Tian et al., 13 Aug 2025).

The paper explicitly identifies Attribute-Relationship Ambiguity as the hardest category overall. This suggests that the most difficult cases are those in which the model must decide not just which field or which edge is intended, but which representational layer of the graph should supply the answer.

3. Benchmark construction and gold answers

AmbiGraph-Eval is built in two stages: data collection and human review (Tian et al., 13 Aug 2025). During data collection, ambiguous examples are obtained through three complementary routes. The first is direct extraction from graph databases, where inherently ambiguous structures or incomplete specifications are identified and turned into ambiguous questions. The second is synthesis from unambiguous data using LLMs, introducing semantic overlaps in attributes and relationships. The third is full generation using LLMs, prompted to create new ambiguous cases by generalizing from validated seed examples (Tian et al., 13 Aug 2025).

The second stage is a multi-round expert validation procedure. Experts review the natural-language question, the graph structure, and the candidate reference answers to ensure that the ambiguity is genuine and that the proposed answers are valid. The process includes multiple rounds of validation, voting, and removal of disputed items. Items under disagreement are discarded, producing a final benchmark intended to contain only expert-verified ambiguity cases (Tian et al., 13 Aug 2025).

The final benchmark contains 560 examples, distributed across the six subcategories as follows (Tian et al., 13 Aug 2025):

Category	Samples	Ambig/Unambig
Attribute, same-entity	86	50/36
Attribute, cross-entity	101	65/36
Relationship, same-entity	86	49/37
Relationship, cross-entity	101	56/45
Attribute-Relationship, same-entity	94	57/37
Attribute-Relationship, cross-entity	92	59/33

Each benchmark item pairs an ambiguous natural-language query with a graph database sample and human-labeled Cypher queries as reference answers. Crucially, the gold answer is not always a single query. For ambiguous cases, the reference is a set of valid interpretations, each associated with a Cypher query returning one valid target result (Tian et al., 13 Aug 2025). This design makes the benchmark closer to ambiguity-aware semantic evaluation in graph-to-text and semantic generation work, where a single source representation may license multiple acceptable outputs and single-reference evaluation is too coarse (Manning et al., 2020).

4. AREA: execution-based ambiguity resolution metric

AmbiGraph-Eval introduces Ambiguity-Resolved Execution Accuracy (AREA) as its principal metric (Tian et al., 13 Aug 2025). AREA is designed to measure whether a model has not only generated executable Cypher, but also recovered all valid interpretations of an ambiguous request.

The target answer set is denoted as

$R_{\text{target}} = \{ t_1, t_2, \dots, t_k \},$

and the model produces Cypher queries $C_1, C_2, \dots, C_n$, whose execution results are $R(C_1), R(C_2), \dots, R(C_n)$ (Tian et al., 13 Aug 2025).

A generated query is valid only if it returns exactly one result and that result belongs to the target set:

$$g(R(C_i)) = \begin{cases} 1 & \text{if } |R(C_i)| = 1 \text{ and } R(C_i) \subseteq R_{\text{target}} \ 0 & \text{otherwise} \end{cases}$$

Coverage is then checked by

$$h\left(\{R(C_i)\}_{i=1}^{n}, R_{\text{target}}\right) = \begin{cases} 1 & \text{if } \bigcup_{i=1}^{n} R(C_i) = R_{\text{target}} \ 0 & \text{otherwise} \end{cases}$$

The per-example score is

$$AREA = h(\{R(C_1), R(C_2), \dots, R(C_n)\}, R_{\text{target}})$$

and the aggregate benchmark score is

$P_{\text{AREA}} = \frac{1}{N}\sum_{i=1}^{N} AREA_i$

(Tian et al., 13 Aug 2025).

AREA differs from ordinary exact-match or standard execution accuracy because it explicitly rewards multi-query ambiguity resolution. A model that generates one correct interpretation but omits another receives no credit for full ambiguity resolution. This makes AREA structurally similar in spirit to evaluation frameworks that separate semantic adequacy from superficial correctness and that treat faithful coverage of the meaning space as primary rather than optional (Manning et al., 2020).

A likely misconception is that AREA is simply execution accuracy on ambiguous queries. It is not. The metric is stricter: each generated query must isolate exactly one valid target result, and the union of all generated results must exactly equal the gold target set (Tian et al., 13 Aug 2025). The benchmark therefore measures ambiguity recognition, interpretation enumeration, Cypher generation, and execution correctness jointly.

5. Evaluation setup and empirical findings

The paper evaluates 9 representative LLMs in a zero-shot strategy with Cypher as the target query language (Tian et al., 13 Aug 2025). The reported models are:

• GPT-4o
• LLaMA-3.1-Instruct-405B
• O1-mini
• Qwen-Plus
• Qwen-2.5-Instruct-72B
• Claude-3.5-Sonnet
• DeepSeek-V2.5
• LLaMA-3.1-Instruct-8B
• Mistral-7B-Instruct-V0.3

The main reported metric is overall average $P_{\text{AREA}}$. The overall ranking is as follows (Tian et al., 13 Aug 2025):

Model	Overall average $P_{\text{AREA}}$
GPT-4o	76.79
LLaMA-3.1-Instruct-405B	76.25
O1-mini	73.39
Qwen-Plus	69.82
Qwen-2.5-Instruct-72B	68.21
Claude-3.5-Sonnet	64.64
DeepSeek-V2.5	61.61
LLaMA-3.1-Instruct-8B	32.86
Mistral-7B-Instruct-V0.3	20.36

The headline result is that even the strongest systems remain well below perfect ambiguity resolution (Tian et al., 13 Aug 2025). The top proprietary and top large open model are close, but no model is robust across all ambiguity categories.

Performance varies substantially by ambiguity type. Attribute Ambiguity is comparatively easier for strong models, with category averages of 80.64 for GPT-4o and 79.78 for O1-mini (Tian et al., 13 Aug 2025). Relationship Ambiguity is handled surprisingly well by some large open models, with LLaMA-3.1-Instruct-405B achieving 85.56 and Qwen-2.5-Instruct-72B reaching 83.42 in category average (Tian et al., 13 Aug 2025). Attribute-Relationship Ambiguity is the most difficult category overall, with top category averages falling to 68.28 for GPT-4o and 64.51 for LLaMA-3.1-Instruct-405B (Tian et al., 13 Aug 2025).

The Same-Entity versus Cross-Entity contrast is one of the more notable findings. Cross-Entity cases are not uniformly harder. In several settings, Same-Entity ambiguity is more difficult, particularly when ambiguity is concentrated in local schema grounding. For example, GPT-4o scores 75.58 on same-entity attribute ambiguity versus 88.19 on cross-entity attribute ambiguity; on relationship ambiguity it scores 65.12 for same-entity versus 92.08 for cross-entity; and on attribute-relationship ambiguity it scores 53.19 for same-entity versus 83.70 for cross-entity (Tian et al., 13 Aug 2025). This suggests that local representational ambiguity within a single entity can be more treacherous than comparative reasoning across entities.

The paper also concludes that reasoning-oriented models do not necessarily have an advantage. O1-mini performs strongly in some localized cases but does not dominate the benchmark. The authors argue that general reasoning strength alone is insufficient; performance depends more on graph ambiguity understanding and Cypher generation competence (Tian et al., 13 Aug 2025).

6. Failure modes, interpretation, and relation to adjacent evaluation work

The paper identifies five main failure modes: detecting ambiguous intent, generating syntactically valid Cypher Query Language, interpreting task instructions and graph structures, performing numerical aggregations, and correctly prioritizing comparisons (Tian et al., 13 Aug 2025). The two primary bottlenecks are explicitly stated to be ambiguity detection and query syntax generation.

These failure modes are consistent with the structure of the benchmark. A model may fail because it does not realize multiple interpretations are possible; because it recognizes ambiguity but emits only one query; because it generates non-executable Cypher; or because it chooses a semantically plausible but incomplete answer set. Questions involving “best,” “highest,” and review-derived comparisons are especially error-prone because they compound schema ambiguity with aggregation ambiguity (Tian et al., 13 Aug 2025).

In the broader context of evaluation methodology, AmbiGraph-Eval belongs to a family of benchmarks that reject single-output scoring for semantically underdetermined tasks. In AMR-to-English generation, referenceless human adequacy evaluation was proposed precisely because one semantic graph may license multiple valid realizations and single-reference metrics can penalize correct paraphrases (Manning et al., 2020). In AMR graph comparison, alignment-free fragment metrics such as SemBleu were proposed to avoid brittle node-alignment search and to reward preservation of larger semantic fragments rather than arbitrary local overlaps (Song et al., 2019). AmbiGraph-Eval applies a related principle to text-to-Cypher: ambiguity-aware evaluation should reward correct coverage of multiple valid interpretations, not just one executable output (Tian et al., 13 Aug 2025).

A plausible implication is that AREA functions for graph semantic parsing much as multi-dimensional human evaluation functions for graph-to-text generation: it distinguishes systems that merely commit confidently from systems that correctly represent the underlying ambiguity. This interpretation is not stated as such in the paper, but it follows from the benchmark design and from its emphasis on interpretation coverage rather than single-query exactness.

The benchmark also contrasts with graph-based evaluation frameworks that decompose objects into structured units and propagate scores over graph representations, such as GraphEval for idea evaluation (Feng et al., 16 Mar 2025), or compare graph abstractions via node- and path-level alignment, as in DiagramEval for SVG diagrams (Liang et al., 29 Oct 2025). Those frameworks are relevant at the level of evaluation philosophy—decomposition, structure sensitivity, and explainable error categories—but AmbiGraph-Eval is specifically an ambiguity benchmark for executable graph query generation rather than a general graph similarity metric (Feng et al., 16 Mar 2025, Liang et al., 29 Oct 2025).

7. Contributions, limitations, and research implications

AmbiGraph-Eval makes three principal contributions. First, it formulates ambiguity in graph query generation as a distinct semantic parsing problem and organizes it into a taxonomy of Attribute Ambiguity, Relationship Ambiguity, and Attribute-Relationship Ambiguity, each split into Same-Entity and Cross-Entity forms (Tian et al., 13 Aug 2025). Second, it introduces a benchmark of 560 expert-validated examples built from real graph structures and LLM-assisted synthesis (Tian et al., 13 Aug 2025). Third, it proposes AREA, an execution-based metric that evaluates whether models recover the full set of valid interpretations rather than a single preferred answer (Tian et al., 13 Aug 2025).

The benchmark has several clear limitations. It is focused exclusively on Cypher/CQL rather than multiple graph query languages. The evaluation is zero-shot only and does not study few-shot prompting, self-consistency, interactive clarification, or tool-augmented generation (Tian et al., 13 Aug 2025). The paper does not present extensive ablations or alternative baselines beyond the multi-model comparison. It also does not explore richer graph-query settings such as multi-hop reasoning in depth, though it explicitly suggests those as future directions (Tian et al., 13 Aug 2025).

The implications for future work are direct. The paper argues for specialized ambiguity-resolution techniques for graph semantic parsing, including syntax-aware prompting and explicit ambiguity highlighting (Tian et al., 13 Aug 2025). More broadly, it suggests that future systems may require a pipeline with ambiguity detection before query generation, explicit production of multiple candidate interpretations, graph-schema-aware grounding, verification of Cypher executability, and possibly interactive clarification when ambiguity cannot be safely resolved automatically.

In that sense, AmbiGraph-Eval establishes ambiguous graph query generation as a distinct benchmark category rather than a minor variant of standard text-to-query parsing. Its central result is not merely that current models make errors, but that ambiguity in graph-structured environments exposes a structural weakness: models often produce a valid query where the task actually demands a valid set of queries (Tian et al., 13 Aug 2025). This shift from single-answer correctness to interpretation coverage is the benchmark’s defining conceptual contribution.