Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis

Abstract: Natural language interfaces to tabular data must handle ambiguities inherent to queries. Instead of treating ambiguity as a deficiency, we reframe it as a feature of cooperative interaction, where the responsibility of query specification is shared among the user and the system. We develop a principled framework distinguishing cooperative queries, i.e., queries that yield a resolvable interpretation, from uncooperative queries that cannot be resolved. Applying the framework to evaluations for tabular question answering and analysis, we analyze the queries in 15 popular datasets, and observe an uncontrolled mixing of query types neither adequate for evaluating a system's execution accuracy nor for evaluating interpretation capabilities. Our framework and analysis of queries shifts the perspective from fixing ambiguity to embracing cooperation in resolving queries. This reflection enables more informed design and evaluation for natural language interfaces for tabular data, for which we outline implications and directions for future research.

• The paper introduces a cooperative query framework that leverages Grice’s principles to differentiate between cooperative and uncooperative queries.
• It reveals that prevalent tabular QA benchmarks are skewed by data-privileged queries, which conflate execution accuracy with interpretability and robustness.
• The study recommends stratified evaluation protocols and user-centric dataset designs to enhance system reliability in real-world data analysis.

Reframing Ambiguity: Cooperative Natural Language Queries for Tabular Data Analysis

Introduction

This paper advances a new conceptual framework for handling ambiguity in natural language queries within tabular data analysis. Instead of viewing ambiguity as a technical flaw to be eliminated, the authors treat it as an inherent and necessary feature of user-system cooperation. They formalize distinctions between cooperative and uncooperative queries, analyze the implications for system evaluation, and scrutinize fifteen prominent benchmarks, uncovering foundational issues related to data-privilege and query ambiguity. The authors argue that most current evaluation practices conflate interpretability, execution accuracy, and robustness by failing to appropriately distinguish among query types, resulting in misleading assessments of system capabilities.

Cooperative Query Framework

The core contribution is the cooperative query framework, grounded in Grice's cooperative principle. In this perspective, queries are interpreted as the product of an implicit division of labor: users specify what they can (consistent with their mental models and knowledge), while systems infer the rest either by applying conventions or exercising agency in selective grounding.

A query is cooperative if it provides sufficient specification—either explicitly or through reasonable inference—so at least one valid, actionable interpretation can be derived. Conversely, an uncooperative query is underspecified to a degree that introduces irresolvable ambiguity, where no actionable interpretation is possible.

As queries become more complex—diagnostic, predictive, or prescriptive—the burden on the user to fully specify analytical procedures increases, often beyond practical feasibility. The framework positions underspecification not as a defect, but as a designed affordance that gives the system agency to complete the interaction.

Figure 1: Illustration of queries with varying levels of specification and the process by which cooperative queries are incrementally grounded by the system until an actionable interpretation is obtained.

The actionable interpretation of a query requires grounding two orthogonal axes:

• Procedural Specification: How clearly does the query define the analytical operation, its parameters, and methodology?
• Data Specification: Is the data scope (entities, temporality, domain) sufficiently defined to locate or filter the relevant data unambiguously?

The authors provide a detailed typology (in the Appendix) to operationalize these requirements. This distinction clarifies when a system can resolve a query independently (unambiguous), when selective system inference is needed (cooperative), and when clarification or iterative dialog with the user is required (uncooperative).

Evaluating Tabular Data Systems: The Pitfalls of Ambiguous Benchmarks

The cooperative query framework underpins a new approach to evaluation. Current benchmarks fail to distinguish between:

• Unambiguous queries (single actionable interpretation): Suitable for measuring execution accuracy.
• Cooperative queries (multiple reasonable interpretations, requiring selective system grounding): Suitable for measuring interpretive and default-decision capabilities.
• Uncooperative queries (irresolvable ambiguity): Useful for evaluating system robustness but not for execution or interpretive accuracy.

Furthermore, the authors identify a pervasive issue: data-privileged queries. These are queries that betray knowledge of table structure, schema, or specific values, and therefore cannot occur in true open-domain settings. Benchmarks saturated with such queries provide artificial shortcuts for systems that would not generalize to true user interactions.

Figure 2: Proportional share of data-privileged queries—where the user exploits knowledge of underlying tables or values—across popular tabular QA and analysis datasets.

Empirically, the share of data-privileged queries is alarmingly high in many datasets. Similarly, the fraction of truly unambiguous queries is often small, particularly in complex analytical tasks. This reflects both the inherent challenges users face in specifying such queries and the tendency for benchmarks to expect a single gold answer even when multiple reasonable groundings exist.

These measurement defects lead to conflated metrics where it is impossible to attribute failures to execution, interpretation, or robustness. As a result, claims about system capabilities in the literature must be interpreted with caution unless the query types in the evaluation protocol are clearly disambiguated and non-data-privileged.

Implications for System Design and Benchmarking

The analysis yields actionable recommendations for both system designers and benchmark creators:

• Stratified Evaluation: Datasets should label queries according to level of specification and data-privilege. Stratified metrics can then accurately diagnose a system's interpretive, execution, and robustness capabilities.
• Dataset Construction: New datasets should be constructed via user-centric protocols that avoid leakage of table/content knowledge and encourage the collection of both unambiguous and intentionally ambiguous (but resolvable) queries. Iterative refinement datasets—where progressive clarifications lead from underspecified to actionable queries—are required to rigorously test systems' ability to recognize, flag, and resolve ambiguity through dialog.
• Flexible Evaluation Protocols: Evaluation must move beyond single-gold answer metrics to protocols that accept multiple valid solutions reflecting different reasonable interpretations or system-chosen defaults. For true open-domain tasks, the notion of ground truth must be adjusted to encompass human-aligned conventions, and systems must transparently disclose their interpretive decisions.
• System Disclosure and Cooperation: Systems should proactively surface the selective groundings they apply and allow user intervention. This requires interfaces that go beyond first-best SQL generation to cooperative, explainable AI interaction.

Theoretical and Practical Impact

This work challenges prevailing assumptions about ambiguity, specification, and the nature of “correctness” in tabular data interfaces. By formalizing the role of cooperation and intentional underspecification, the framework provides an analytical tool for dissecting failures, designing more flexible systems, and building meaningfully challenging evaluation resources. The results directly impact the design and deployment of LLM-based agents in open-domain table QA, business analytics, and data science workflows—domains where natural, underspecified language is the norm.

A notable claim from the empirical analyses is that most popular datasets used for evaluating open-domain analytical systems are confounded by high rates of data-privileged queries and fail to cleanly separate interpretation from execution performance—a diagnosis that, if widely acknowledged, would require reevaluation of progress in the field.

Future research directions include automated detection of data-privileged and ambiguous queries, the development of conversion and repair pipelines for existing benchmarks, user simulation studies to model realistic ambiguity, and integration of cooperative dialog agents capable of mixed-initiative disambiguation.

Conclusion

This paper reframes ambiguity in natural language analytical queries from a technical obstacle to a structured affordance for cooperative human-AI interaction. By introducing a rigorous framework for query specification and actionable interpretation, the authors provide conceptual and empirical justification for overhauling both system evaluation and benchmark design. The findings call for a shift towards stratified evaluation protocols, datasets free from data-privilege leakage, flexible acceptance criteria for multiple correct interpretations, and systems designed to transparently reason about, disclose, and negotiate ambiguity. This reorientation is essential for the next generation of robust, user-aligned tabular data analysis agents.