Underspecification does not imply Incoherence: The Risks of Semantic Collapse in Coding Models

Abstract: LLMs have become increasingly effective at generating code when task descriptions are clear and precise. Yet, in practice, user-provided task descriptions are often ambiguous, incomplete, or contradictory, leaving critical aspects of the intended program behavior underspecified. In such cases, multiple behaviorally distinct interpretations may satisfy the description equally well, yet semantically differ in ways that matter/affect the user intent. A natural expectation, often assumed by researchers, is that prompt underspecification manifests as incoherence: When asked multiple times, an LLM produces multiple semantically distinct implementations reflecting the ambiguity of the task description. In this paper, we challenge this assumption. We find that LLMs frequently collapse onto a single incorrect interpretation of the task description, consistently generating coherent but behaviorally misaligned code. We term this failure mode detrimental semantic collapse and find that it affects over 10% of tasks in MBPP, 3% in HumanEval, and 32% of LiveCodeBench, all benchmarks assumed to be well-specified. By deliberately injecting underspecification issues in the benchmark prompts, the rate rises to over 5 times, exposing a fundamental blind spot in disambiguation and correctness estimation techniques that rely on incoherence as a proxy for prompt underspecification.

• The paper demonstrates that underspecification leads to detrimental semantic collapse where coding LLMs converge on a single, incorrect interpretation.
• Comprehensive experiments show that ambiguity, incompleteness, and contradiction drastically increase collapse rates, undermining incoherence-based correctness detection.
• Interactive disambiguation improves pass rates but fails to mitigate collapse in many cases, highlighting the need for new evaluation protocols.

Semantic Collapse in Coding LLMs: Underspecification without Incoherence

Introduction

The paper "Underspecification does not imply Incoherence: The Risks of Semantic Collapse in Coding Models" (2607.01953) investigates the behavioral failure modes of state-of-the-art LLMs in automatic code generation tasks under practical conditions of prompt ambiguity, incompleteness, and contradiction. The work critically addresses the prevalent assumption that prompt underspecification naturally leads LLMs to produce semantically diverse, incoherent outputs, an assumption underpinning much recent research leveraging semantic clustering for specification and correctness assessment. Through comprehensive experiments with leading coding LLMs across canonical benchmarks—including MBPP, HumanEval, and LiveCodeBench, both in original and systematically underspecified forms—the study identifies and quantifies "detrimental semantic collapse": a mode where LLMs converge with high internal consistency on a single, yet incorrect, interpretation, rendering incoherence-based detection fundamentally unreliable for underspecified prompts.

Methodology

The experimental pipeline centers on measuring semantic support in the output distributions of Claude Sonnet 4.5, GPT-4.1-mini, and Qwen3-32B. For a prompt $d$ that is ambiguous or otherwise underspecified (i.e., admits multiple plausible ground truths), repeated stochastic decoding (temperature $\tau$ up to 1) is performed, and generations are semantically clustered based on execution on gold or LLM-generated test suites. Semantic collapse is operationally detected when all $k$ generated programs (up to $k=25$ per task) fall into a single equivalence class under this behavioral metric. Collapse is termed "detrimental" when the collapsed cluster does not match the reference implementation, as determined by the original test suite.

The benchmarks are manipulated to introduce three primary types of underspecification: (i) ambiguity (admitting mutually reasonable, conflicting interpretations), (ii) incompleteness (missing requirements), and (iii) contradiction (internally inconsistent directions). For each, the rate of semantic collapse (and specifically detrimental collapse) is measured alongside pass@k and inter-sample inconsistency, both in isolation and under simulated interactive clarification dialogues.

Empirical Findings

Detrimental Semantic Collapse Is Prevalent and Exacerbated by Underspecification

A key observation is that even in canonical, "well-specified" benchmarks, detrimental collapse is already material: 10--16% of MBPP tasks, 3% of HumanEval tasks, and 18--32% of LiveCodeBench tasks show this failure mode, where models confidently commit to a consistently incorrect interpretation without any surface indication or sample-level diversity.

Figure 2: Detrimental semantic collapse on MBPP quantifies the fraction of tasks where LLMs generate only erroneous programs, highlighting the silent failure mode.

Under systematically injected underspecification, the frequency of detrimental collapse increases dramatically. In MBPP variants, the rate reaches 23--55%, depending on underspecification type and model, with a maximum 5.53x increase over the original specification. This trend holds, albeit at different scales, in HumanEval and LiveCodeBench.

Figure 4: LLM-generated test inputs, when used for semantic clustering, may further increase detrimental collapse, as generated tests align more closely to LLM priors.

Notably, when test inputs are generated by LLMs conditioned on ambiguous prompts (as opposed to using gold test cases), the detrimental collapse rate can rise further, especially for certain task sets. This indicates that the model's internal understanding of the underspecified prompt is reinforced both during code generation and during test generation.

Absence of Incoherence Is Not a Reliable Signal for Correctness

While the average trend is for inconsistency (i.e., inter-sample semantic diversity) to rise somewhat on underspecified prompts, a majority of underspecified tasks nevertheless exhibit perfect or near-perfect collapse, suggesting LLMs are over-regularized toward one dominant decision and do not adequately represent or explore the feasible semantics space.

The critical implication is that—contrary to widespread practice in prompt quality detection and correctness estimation—lack of incoherence cannot be used as an indicator of sufficient or correct task understanding. This is most damaging for techniques that rely on clustering for semi-automatic prompt disambiguation, oracle-free correctness estimation, or test selection.

Interactive Disambiguation Only Partially Alleviates the Problem

Simulation of multi-turn clarification dialogues (inspired by systems like ClarifyGPT) shows that, while pass rates improve with every clarifying question, a substantial proportion of tasks affected by detrimental collapse see no benefit: these prompts are never flagged for clarification, and the systems return a single incorrect program with high confidence.

Figure 6: Clarification-driven improvement in MBPP-Ambiguous: the pass rate increases with T consultations, but a portion of erroneous, semantically collapsed tasks remains immune to improvement.

Tables also show that even after multiple clarification turns, the gap in final pass rates relative to the original, well-specified benchmarks remains substantial on benchmarks like MBPP. This is consistent with collapse being a pre-interaction, model-internal process: disambiguation only helps when initial generations are diverse enough to trigger clarification.

Implications for Evaluation and Deployed Systems

The paper demonstrates that detrimental semantic collapse is not a minor or easily mitigated edge-case, but a statistical inevitability in LLM-based code generation whenever underspecification is present and the model lacks an explicit mechanism to surface uncertainty. This undercuts the reliability of self-consistency [0002WSLCNCZ23], execution similarity and clustering-based selection [(Li et al., 2022), ShiFGZW22], and recent work on incoherence-driven correctness assignment (Sharma et al., 17 Feb 2025, Zhang et al., 31 Mar 2026, Launer et al., 17 Apr 2026). The cost-accuracy tradeoffs in increasing $k$ (generation count) or $\tau$ (temperature) provide only limited mitigation; coverage of rare interpretations often implies infeasible computational expense, especially for larger and more complex codebases.

Figure 1: Claude Sonnet 4.5 exhibits pronounced detrimental collapse on underspecified MBPP tasks, typical among state-of-the-art code models.

For researchers, this necessitates the development of new evaluation protocols that move beyond observable output diversity, incorporating explicit uncertainty elicitation, analysis of internal traces, and alternative prompting or joint reasoning strategies. For practitioners deploying LLMs in code-critical scenarios, over-reliance on sample diversity or confidence as a proxy for correctness under ambiguous requirements leads to unrecognized failures—an unacceptable risk in production environments.

Conclusion

This work discredits the tacit assumption underlying a large body of LLM code evaluation research: that inconsistency is a necessary byproduct of underspecification or model misunderstanding. Empirically, LLMs in practice manifest a detrimental form of internal coherence—semantic collapse—producing unanimous, but incorrect, programs. The absence of observable diversity does not guarantee either sufficient specification or correctness, fundamentally limiting the utility of clustering-based evaluation, candidate selection, and prompt disambiguation.

Future research must address theoretical and practical approaches for direct ambiguity detection, richer uncertainty estimates, and mechanisms for controlled semantic exploration. The challenge is especially acute in realistic, project-scale codebases, where underspecification is the norm and the cost of failure is high.

---

References

The essay is based on "Underspecification does not imply Incoherence: The Risks of Semantic Collapse in Coding Models" (2607.01953) and related foundational work cited therein.