---
title: 'AmbiGraph-Eval: Benchmark Ambiguity in Graph Queries'
url: https://www.emergentmind.com/topics/ambigraph-eval
type: topic
---

# AmbiGraph-Eval: Benchmark Ambiguity in Graph Queries

Searching arXiv for AmbiGraph-Eval and closely related evaluation papers.
AmbiGraph-Eval is a benchmark for evaluating whether large language models can correctly handle ambiguity in natural-language graph queries, specifically in text-to-Cypher semantic parsing. It targets cases in which a user request is plausibly interpretable in multiple ways and where graph-structured data amplifies the ambiguity because meaning may reside in node attributes, edge types, relationship attributes, or graph neighborhoods rather than in a single flat schema location. The benchmark introduces a taxonomy of graph-query ambiguities, a curated dataset of ambiguous and unambiguous examples paired with expert-verified Cypher answers, and an execution-based metric designed to reward models that recover all valid interpretations rather than committing to only one plausible reading [2508.09631].

## 1. Problem formulation and scope

AmbiGraph-Eval studies ambiguity in **text-to-Cypher / text-to-CQL** graph query generation. The central task is not merely to translate natural language into an executable query, but to do so when the natural-language request admits multiple valid interpretations grounded in the same graph schema. In this setting, a model must recognize that the ambiguity is genuine, map each interpretation to the graph correctly, and generate Cypher queries whose execution results collectively cover the full target answer set [2508.09631].

The benchmark is motivated by the claim that ambiguity in graph semantic parsing is more difficult than in SQL-oriented settings because graph queries require not only schema grounding but also choice of traversal path, relationship semantics, comparison basis, aggregation scope, and syntactically correct Cypher realization. A phrase such as “best evaluated restaurant” can be ambiguous both at the level of semantic grounding and at the level of aggregation: the graph may contain multiple rating-like signals, and “best” may refer either to an individual maximum or to an aggregate across related nodes. This makes graph ambiguity simultaneously a problem of lexical grounding, relational interpretation, and query-program synthesis [2508.09631].

The paper further argues that modern LLM-based semantic parsers often exhibit **self-preference bias** in such cases: instead of signaling that multiple interpretations are valid, they tend to commit to one reading and emit a single precise Cypher query. A plausible implication is that the benchmark is intended as a stress test not only for semantic parsing accuracy but also for ambiguity detection and interpretation coverage.

## 2. Ambiguity taxonomy

AmbiGraph-Eval organizes graph-query ambiguity into **three primary ambiguity types**, each divided into **Same-Entity** and **Cross-Entity** cases, yielding six subcategories in total [2508.09631].

| Primary type | Same-Entity | Cross-Entity |
|---|---|---|
| Attribute Ambiguity | Multiple plausible attributes for one entity | Unclear attribute basis for comparison across entities |
| Relationship Ambiguity | Multiple plausible relationships for one entity | Unclear relation-derived comparison criterion across entities |
| Attribute-Relationship Ambiguity | Unclear whether answer comes from an attribute or from relationships for one entity | Unclear whether comparison should use attributes or relational evidence across entities |

**Attribute Ambiguity** occurs when a request refers to an entity property but multiple graph attributes could satisfy it. The representative example is asking for the **citation count** of a paper when the graph stores citation counts from multiple sources such as **Google Scholar** and **Semantic Scholar**. In the Same-Entity variant, the ambiguity is local to one entity. In the Cross-Entity variant, the ambiguity affects the basis of comparison, as in “Which paper has the highest citation count?” [2508.09631].

**Relationship Ambiguity** arises when the intended answer depends on the entity’s graph relationships, but the relevant relationship is underspecified. The paper’s examples include an architect’s **“contributions”**, which may correspond to different relation types, and “Who is the best doctor according to patient reviews?”, where the relevant review-derived signal is unclear. Same-Entity cases require choosing among multiple relations attached to one entity; Cross-Entity cases require choosing the correct comparison standard derived from relationships [2508.09631].

**Attribute-Relationship Ambiguity** is the most graph-specific category. Here, the ambiguity is whether the answer should be obtained from a node’s own property or from related entities or relationships. The canonical example is “What is the rating for the movie Inception?”, which may refer either to an intrinsic property such as **IMDb rating** or to review information reachable through graph traversal. The Cross-Entity version extends this to comparative questions such as “Which city hall has the highest evaluation?”, where both the signal source and the comparison procedure are ambiguous [2508.09631].

The paper explicitly identifies **Attribute-Relationship Ambiguity** as the hardest category overall. This suggests that the most difficult cases are those in which the model must decide not just *which field* or *which edge* is intended, but *which representational layer of the graph* should supply the answer.

## 3. Benchmark construction and gold answers

AmbiGraph-Eval is built in two stages: **data collection** and **human review** [2508.09631]. During data collection, ambiguous examples are obtained through three complementary routes. The first is **direct extraction from graph databases**, where inherently ambiguous structures or incomplete specifications are identified and turned into ambiguous questions. The second is **synthesis from unambiguous data using LLMs**, introducing semantic overlaps in attributes and relationships. The third is **full generation using LLMs**, prompted to create new ambiguous cases by generalizing from validated seed examples [2508.09631].

The second stage is a multi-round expert validation procedure. Experts review the natural-language question, the graph structure, and the candidate reference answers to ensure that the ambiguity is genuine and that the proposed answers are valid. The process includes multiple rounds of validation, voting, and removal of disputed items. Items under disagreement are discarded, producing a final benchmark intended to contain only expert-verified ambiguity cases [2508.09631].

The final benchmark contains **560** examples, distributed across the six subcategories as follows [2508.09631]:

| Category | Samples | Ambig/Unambig |
|---|---:|---:|
| Attribute, same-entity | 86 | 50/36 |
| Attribute, cross-entity | 101 | 65/36 |
| Relationship, same-entity | 86 | 49/37 |
| Relationship, cross-entity | 101 | 56/45 |
| Attribute-Relationship, same-entity | 94 | 57/37 |
| Attribute-Relationship, cross-entity | 92 | 59/33 |

Each benchmark item pairs an ambiguous natural-language query with a graph database sample and **human-labeled Cypher queries** as reference answers. Crucially, the gold answer is not always a single query. For ambiguous cases, the reference is a **set of valid interpretations**, each associated with a Cypher query returning one valid target result [2508.09631]. This design makes the benchmark closer to ambiguity-aware semantic evaluation in graph-to-text and semantic generation work, where a single source representation may license multiple acceptable outputs and single-reference evaluation is too coarse [2004.06814].

## 4. AREA: execution-based ambiguity resolution metric

AmbiGraph-Eval introduces **Ambiguity-Resolved Execution Accuracy (AREA)** as its principal metric [2508.09631]. AREA is designed to measure whether a model has not only generated executable Cypher, but also recovered all valid interpretations of an ambiguous request.

The target answer set is denoted as
$$
R_{\text{target}} = \{ t_1, t_2, \dots, t_k \},
$$
and the model produces Cypher queries \(C_1, C_2, \dots, C_n\), whose execution results are \(R(C_1), R(C_2), \dots, R(C_n)\) [2508.09631].

A generated query is valid only if it returns exactly one result and that result belongs to the target set:
$$
g(R(C_i)) =
\begin{cases}
1 & \text{if } |R(C_i)| = 1 \text{ and } R(C_i) \subseteq R_{\text{target}} \\
0 & \text{otherwise}
\end{cases}
$$

Coverage is then checked by
$$
h\left(\{R(C_i)\}_{i=1}^{n}, R_{\text{target}}\right) =
\begin{cases}
1 & \text{if } \bigcup_{i=1}^{n} R(C_i) = R_{\text{target}} \\
0 & \text{otherwise}
\end{cases}
$$

The per-example score is
$$
AREA = h(\{R(C_1), R(C_2), \dots, R(C_n)\}, R_{\text{target}})
$$
and the aggregate benchmark score is
$$
P_{\text{AREA}} = \frac{1}{N}\sum_{i=1}^{N} AREA_i
$$
[2508.09631].

AREA differs from ordinary exact-match or standard execution accuracy because it explicitly rewards **multi-query ambiguity resolution**. A model that generates one correct interpretation but omits another receives no credit for full ambiguity resolution. This makes AREA structurally similar in spirit to evaluation frameworks that separate semantic adequacy from superficial correctness and that treat faithful coverage of the meaning space as primary rather than optional [2004.06814].

A likely misconception is that AREA is simply execution accuracy on ambiguous queries. It is not. The metric is stricter: each generated query must isolate exactly one valid target result, and the union of all generated results must exactly equal the gold target set [2508.09631]. The benchmark therefore measures ambiguity recognition, interpretation enumeration, Cypher generation, and execution correctness jointly.

## 5. Evaluation setup and empirical findings

The paper evaluates **9 representative LLMs** in a **zero-shot strategy** with **Cypher** as the target query language [2508.09631]. The reported models are:

- **GPT-4o**
- **LLaMA-3.1-Instruct-405B**
- **O1-mini**
- **Qwen-Plus**
- **Qwen-2.5-Instruct-72B**
- **Claude-3.5-Sonnet**
- **DeepSeek-V2.5**
- **LLaMA-3.1-Instruct-8B**
- **Mistral-7B-Instruct-V0.3**

The main reported metric is overall average \(P_{\text{AREA}}\). The overall ranking is as follows [2508.09631]:

| Model | Overall average \(P_{\text{AREA}}\) |
|---|---:|
| GPT-4o | 76.79 |
| LLaMA-3.1-Instruct-405B | 76.25 |
| O1-mini | 73.39 |
| Qwen-Plus | 69.82 |
| Qwen-2.5-Instruct-72B | 68.21 |
| Claude-3.5-Sonnet | 64.64 |
| DeepSeek-V2.5 | 61.61 |
| LLaMA-3.1-Instruct-8B | 32.86 |
| Mistral-7B-Instruct-V0.3 | 20.36 |

The headline result is that even the strongest systems remain well below perfect ambiguity resolution [2508.09631]. The top proprietary and top large open model are close, but no model is robust across all ambiguity categories.

Performance varies substantially by ambiguity type. **Attribute Ambiguity** is comparatively easier for strong models, with category averages of **80.64** for GPT-4o and **79.78** for O1-mini [2508.09631]. **Relationship Ambiguity** is handled surprisingly well by some large open models, with **LLaMA-3.1-Instruct-405B** achieving **85.56** and **Qwen-2.5-Instruct-72B** reaching **83.42** in category average [2508.09631]. **Attribute-Relationship Ambiguity** is the most difficult category overall, with top category averages falling to **68.28** for GPT-4o and **64.51** for LLaMA-3.1-Instruct-405B [2508.09631].

The Same-Entity versus Cross-Entity contrast is one of the more notable findings. Cross-Entity cases are not uniformly harder. In several settings, **Same-Entity ambiguity is more difficult**, particularly when ambiguity is concentrated in local schema grounding. For example, GPT-4o scores **75.58** on same-entity attribute ambiguity versus **88.19** on cross-entity attribute ambiguity; on relationship ambiguity it scores **65.12** for same-entity versus **92.08** for cross-entity; and on attribute-relationship ambiguity it scores **53.19** for same-entity versus **83.70** for cross-entity [2508.09631]. This suggests that local representational ambiguity within a single entity can be more treacherous than comparative reasoning across entities.

The paper also concludes that **reasoning-oriented models do not necessarily have an advantage**. O1-mini performs strongly in some localized cases but does not dominate the benchmark. The authors argue that general reasoning strength alone is insufficient; performance depends more on graph ambiguity understanding and Cypher generation competence [2508.09631].

## 6. Failure modes, interpretation, and relation to adjacent evaluation work

The paper identifies five main failure modes: **detecting ambiguous intent**, **generating syntactically valid Cypher Query Language**, **interpreting task instructions and graph structures**, **performing numerical aggregations**, and **correctly prioritizing comparisons** [2508.09631]. The two primary bottlenecks are explicitly stated to be **ambiguity detection** and **query syntax generation**.

These failure modes are consistent with the structure of the benchmark. A model may fail because it does not realize multiple interpretations are possible; because it recognizes ambiguity but emits only one query; because it generates non-executable Cypher; or because it chooses a semantically plausible but incomplete answer set. Questions involving “best,” “highest,” and review-derived comparisons are especially error-prone because they compound schema ambiguity with aggregation ambiguity [2508.09631].

In the broader context of evaluation methodology, AmbiGraph-Eval belongs to a family of benchmarks that reject single-output scoring for semantically underdetermined tasks. In AMR-to-English generation, referenceless human adequacy evaluation was proposed precisely because one semantic graph may license multiple valid realizations and single-reference metrics can penalize correct paraphrases [2004.06814]. In AMR graph comparison, alignment-free fragment metrics such as SemBleu were proposed to avoid brittle node-alignment search and to reward preservation of larger semantic fragments rather than arbitrary local overlaps [1905.10726]. AmbiGraph-Eval applies a related principle to text-to-Cypher: ambiguity-aware evaluation should reward correct coverage of multiple valid interpretations, not just one executable output [2508.09631].

A plausible implication is that AREA functions for graph semantic parsing much as multi-dimensional human evaluation functions for graph-to-text generation: it distinguishes systems that merely commit confidently from systems that correctly represent the underlying ambiguity. This interpretation is not stated as such in the paper, but it follows from the benchmark design and from its emphasis on interpretation coverage rather than single-query exactness.

The benchmark also contrasts with graph-based evaluation frameworks that decompose objects into structured units and propagate scores over graph representations, such as GraphEval for idea evaluation [2503.12600], or compare graph abstractions via node- and path-level alignment, as in DiagramEval for SVG diagrams [2510.25761]. Those frameworks are relevant at the level of evaluation philosophy—decomposition, structure sensitivity, and explainable error categories—but AmbiGraph-Eval is specifically an ambiguity benchmark for executable graph query generation rather than a general graph similarity metric [2503.12600; 2510.25761].

## 7. Contributions, limitations, and research implications

AmbiGraph-Eval makes three principal contributions. First, it formulates ambiguity in graph query generation as a distinct semantic parsing problem and organizes it into a taxonomy of **Attribute Ambiguity**, **Relationship Ambiguity**, and **Attribute-Relationship Ambiguity**, each split into Same-Entity and Cross-Entity forms [2508.09631]. Second, it introduces a benchmark of **560** expert-validated examples built from real graph structures and LLM-assisted synthesis [2508.09631]. Third, it proposes **AREA**, an execution-based metric that evaluates whether models recover the full set of valid interpretations rather than a single preferred answer [2508.09631].

The benchmark has several clear limitations. It is focused exclusively on **Cypher/CQL** rather than multiple graph query languages. The evaluation is **zero-shot only** and does not study few-shot prompting, self-consistency, interactive clarification, or tool-augmented generation [2508.09631]. The paper does not present extensive ablations or alternative baselines beyond the multi-model comparison. It also does not explore richer graph-query settings such as multi-hop reasoning in depth, though it explicitly suggests those as future directions [2508.09631].

The implications for future work are direct. The paper argues for **specialized ambiguity-resolution techniques** for graph semantic parsing, including **syntax-aware prompting** and **explicit ambiguity highlighting** [2508.09631]. More broadly, it suggests that future systems may require a pipeline with ambiguity detection before query generation, explicit production of multiple candidate interpretations, graph-schema-aware grounding, verification of Cypher executability, and possibly interactive clarification when ambiguity cannot be safely resolved automatically.

In that sense, AmbiGraph-Eval establishes ambiguous graph query generation as a distinct benchmark category rather than a minor variant of standard text-to-query parsing. Its central result is not merely that current models make errors, but that ambiguity in graph-structured environments exposes a structural weakness: models often produce a valid query where the task actually demands a valid *set* of queries [2508.09631]. This shift from single-answer correctness to interpretation coverage is the benchmark’s defining conceptual contribution.

Source: https://www.emergentmind.com/topics/ambigraph-eval