ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation

Abstract: LLMs have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input requirements, which are frequently ambiguous, incomplete, or underspecified. While LLMs excel at one-shot code synthesis, their ability to proactively clarify intent remains underexplored, as a critical trait for robust software engineering. Existing benchmarks largely overlook this interactive bottleneck, assuming perfectly specified prompts that do not reflect the iterative nature of requirement elicitation. To bridge this gap, we introduce ClarifyCodeBench, a novel interactive benchmark for evaluating LLMs' capability in resolving requirement ambiguity. Constructed from real-world programming tasks, ClarifyCodeBench features high-quality manual annotations, including N unique ambiguity types, associated clarification questions, and corresponding ground-truth answers. Furthermore, we formalize two rigorous metrics to assess the interaction quality: Turn-discounted Key Question Rate, which penalizes inefficient questioning, and Optimal Round Adherence, which measures the precision of the elicitation process. We conduct a systematic evaluation of six state-of-the-art LLMs using ClarifyCodeBench. Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to effective requirement clarification; 2) The Reasoning Paradox: While increased computational thinking enhances code correctness, it yields marginal gains in identifying ambiguities; 3) The Multi-ambiguity Ceiling: LLMs' clarification performance degrades sharply as the density of ambiguities increases, revealing a significant bottleneck in handling complex, real-world specifications. Our work underscores the necessity for future AI4SE research to transition from static synthesis to interactive elicitation.

• The paper introduces ClarifyCodeBench, a benchmark that tests LLMs’ performance in clarifying ambiguous code generation requirements through interactive evaluation metrics.
• It employs controlled ambiguous tasks with detailed taxonomy and quantifies clarification using TKQR and ORA metrics, reflecting real-world software development challenges.
• Empirical results reveal that state-of-the-art LLMs struggle with multi-ambiguity resolution, with functional correctness dropping by up to 19.8 percentage points.

ClarifyCodeBench: A Benchmark for Evaluating LLM Clarification of Ambiguous Code Generation Requirements

Introduction

LLMs have shown significant ability in code synthesis, but robust deployment in software engineering remains challenging due to the frequent presence of ambiguous or underspecified requirements. Existing benchmarks for code generation predominantly assume clear, fully specified prompts and fail to assess the ability of LLMs to recognize or resolve ambiguities prior to code synthesis. "ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation" (2607.00711) introduces a benchmark designed to directly evaluate requirement clarification capabilities in LLMs, reflecting the interactive and iterative nature of real-world software development.

Figure 1: An ambiguous code generation requirement illustrating multiple plausible interpretations and the challenge of intent disambiguation.

Benchmark Design and Taxonomy

ClarifyCodeBench is constructed on top of LiveCodeBench, leveraging its complete code generation tasks to introduce controlled ambiguities by deletion-only editing. Each benchmark instance contains:

• An ambiguous requirement (created by omitting critical detail from an otherwise well-formed task)
• One to three annotated key clarification questions and their ground-truth answers (corresponding to each ambiguity point)
• A fine-grained ambiguity type label, grounded in established principles of requirement quality (unambiguity, completeness, verifiability)
• An executable test suite for functional correctness assessment

The ambiguity taxonomy spans ten categories, including Terminology, Behavior, Edge Cases, Indices & Ranges, Ordering & Atomicity, Output Format, Comparison Rules, Units, Collection Semantics, and Numerical Precision.

Figure 2: Distribution of ambiguity types in ClarifyCodeBench—Terminology and Output Format are prominent categories, illustrating the spectrum of requirement ambiguity LLMs must address.

The dataset comprises 419 ambiguous tasks: 199 single-ambiguity, 169 double-ambiguity, and 51 triple-ambiguity instances, supporting evaluation of both basic and compound ambiguity resolution.

Evaluation Protocol and Metrics

To assess interactive clarification, the evaluation loop formalizes the following:

• The LLM, under a structured system prompt, must decide whether to synthesize code or ask exactly one clarification question per turn.
• If a question matches a ground-truth ambiguity point (as adjudicated by an LLM-based judge), it receives the corresponding answer; otherwise, a default irrelevance response is returned.
• This interaction proceeds until the LLM produces code or reaches a maximum round bound.

Figure 3: Overview of the evaluation protocol, specifying the multi-round process combining ambiguity detection, question answering, and final code synthesis.

Clarification is quantified along two axes:

• Turn-discounted Key Question Rate (TKQR): A normalized, DCG-inspired metric rewarding early and unique identification of key clarifying questions.
• Optimal Round Adherence (ORA): A Gaussian penalty-based measure of interaction efficiency, favoring minimal and sufficient rounds matching the annotated ideal.

Functional correctness is measured post-interaction using pass@1.

Empirical Results

Clarification and Code Generation Performance

ClarifyCodeBench reveals several critical findings:

• Clarification Remains a Bottleneck: All evaluated SOTA LLMs (including GPT-4o, GPT-5, Gemini-2.5-Flash, Claude-Sonnet-4.5, DeepSeek-V3.2, and Qwen3-235B-A22B) achieve markedly low TKQR (≤0.30 in best cases) and suboptimal ORA (≤0.50 at best). Functional correctness under ambiguous inputs drops by 7.8–19.8 absolute percentage points compared to complete requirements.
• Reasoning Does Not Translate to Clarification Gains: Integrating explicit reasoning ("thinking" models) improves code correctness on fully specified tasks but confers negligible improvement for ambiguity resolution, with only marginal gains in TKQR and ORA and, in some cases, even a drop in ambiguous-input pass@1.
• Ambiguity Type Matters: LLMs can more reliably resolve explicit, local ambiguities (Units, Numerical Precision) with hit rates >65%, but struggle with semantic/structural ambiguities (Collection Semantics, Comparison Rules, Ordering & Atomicity), where hit rates remain <20%.
• Multi-ambiguity Barrier: When two or more ambiguities are present, the probability of the LLM uncovering all required clarifications collapses; while single-ambiguity identification occurs up to 30% of the time, multi-point resolution drops near zero.
• Shallow Clarification Depth: Interaction depth is consistently below optimum—models ask fewer clarification questions than required and often default to code generation prematurely, especially in the presence of ambiguity compounding.

Figure 4: Pass@1 (\%) for five LLMs under four input settings, illustrating the pronounced performance drop due to ambiguity, and the partial recovery through both interactive and oracle clarification.

Error Analysis

A detailed manual analysis of failure cases uncovers recurring error types induced by unresolved ambiguity:

• Runtime Errors: Caused by omitted constraints, leading to misinterpretation of input or edge-case handling.
• Wrong Answers: Typically result from missing examples or requirements that clarify semantic constraints (e.g., the minimum solution in palindrome construction).
• Time Limit Exceeded: Ambiguity leads to conservative or inefficient implementations (e.g., explicit grid manipulation instead of graph abstraction) that would not arise under a fully specified requirement.

These findings reinforce that lack of clarification alters both the accuracy and the failure modalities of generated code, highlighting the compounding risk in practical software engineering scenarios.

Implications and Future Directions

ClarifyCodeBench exposes a fundamental decoupling between code synthesis prowess and clarification competence in current LLMs. The study defines the following implications and future research directions:

• Richer Agentic Workflows: The current evaluation isolates model-level clarification; integration with tool-use or agentic code development may better reflect real-world resolution capabilities but requires orthogonal benchmarking.
• Data Leakage Mitigation: Ensuring that models have not seen the tasks in training data remains an open issue, as proprietary LLMs may leverage prior exposure to compensate for deficient clarification behavior.
• Augmented Training Pipelines: Significant gains may require clarification-centric pretraining, data augmentation with synthetic ambiguities, or explicitly teaching interactive elicitation protocols.
• Evaluation Beyond ClarifyCodeBench: New task formulations that combine ambiguity, iterative design, and agent interactions will be critical for next-generation AI4SE benchmarks.

Conclusion

ClarifyCodeBench introduces a rigorous, ambiguity-centered benchmark that surfaces the limitations of LLMs in interactive requirement clarification—a critical prerequisite for reliable AI-driven code synthesis. The controlled construction, categorization of ambiguity types, and the introduction of TKQR/ORA metrics collectively offer an incisive diagnostic framework. The empirical results signal that further advances in code generation will hinge not only on model scale or reasoning but also on systematic improvement of ambiguity detection and resolution mechanisms. This benchmark is expected to guide future research efforts targeting robust, interactive AI assistants in software engineering.