When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation

Abstract: LLMs are increasingly used for code generation, yet the correctness of their outputs depends not only on model capability but also on how tasks are specified. Prior studies demonstrate that small changes in natural language prompts, particularly under-specification can substantially reduce code correctness; however, these findings are largely based on minimal-specification benchmarks such as HumanEval and MBPP, where limited structural redundancy may exaggerate sensitivity. In this exploratory study, we investigate how prompt structure, task complexity, and specification richness interact with LLM robustness to prompt mutations. We evaluate 10 different models across HumanEval and the structurally richer LiveCodeBench. Our results reveal that robustness is not a fixed property of LLMs but is highly dependent on prompt structure: the same under-specification mutations that degrade performance on HumanEval have near-zero net effect on LiveCodeBench due to redundancy across descriptions, constraints, examples, and I/O conventions. Surprisingly, we also find that prompt mutations can improve correctness. In LiveCodeBench, under-specification often breaks misleading lexical or structural cues that trigger incorrect retrieval-based solution strategies, leading to correctness improvements that counterbalance degradations. Manual analysis identifies consistent mechanisms behind these improvements, including the disruption of over-fitted terminology, removal of misleading constraints, and elimination of spurious identifier triggers. Overall, our study shows that structurally rich task descriptions can substantially mitigate the negative effects of under-specification and, in some cases, even enhance correctness. We outline categories of prompt modifications that positively influence the behavior of LLM code-generation, offering practical insights for writing robust prompts.

• The paper demonstrates that under-specified prompts improve LLM code correctness by removing misleading retrieval cues that lead to memorized, incorrect solutions.
• It employs a comparative analysis between minimal-specification (HumanEval) and rich-structure (LiveCodeBench) benchmarks across 10 models under varied prompt mutations.
• Findings indicate that lean, redundant prompts enhance semantic reasoning, challenging the assumption that excessive detail always yields better outcomes.

When Prompt Under-Specification Improves Code Correctness in LLM-Based Code Generation

Introduction and Problem Statement

Prompt design heavily impacts the robustness and functional correctness of LLM-based code generation. Although earlier work has established that both ambiguous and under-specified prompts typically degrade model outputs, this conclusion has been largely driven by empirics over minimal-specification datasets such as HumanEval and MBPP. These benchmarks lack orthogonal specification regions, intensifying any observed prompt sensitivity. This paper rigorously analyzes the interaction between prompt structure, specification richness, and LLM robustness to under-specification and mutation.

A distinguishing aspect of this study is the comparative analysis between HumanEval and LiveCodeBench; the latter is structurally richer, encoding constraints, examples, and I/O conventions independently. The investigation spans 10 models, from open-source 6B/7B models to proprietary reasoning systems, under three classes of prompt mutations: Lexical Vagueness (LV), Under-Specification (US), and Syntax/Formatting perturbations (SF). The study advances understanding by identifying mechanisms where prompt under-specification actively improves code correctness, contradicting prevailing assumptions.

Experimental Methodology

The evaluation considers both aggregate correctness (Pass@1) and fine-grained transition analysis (Pass→Fail, Fail→Pass) under prompt mutations. All mutations are generated using LLMs with independent automated and manual validation. The benchmark selection is methodically controlled: HumanEval as a minimal-specification standard and LiveCodeBench as a contamination-free, multi-layered competitive programming suite.

Three mutation classes are defined operationally:

• Lexical Vagueness (LV): Semantic scope reduced by paraphrasing, broadening, or identifier weakening.
• Under-Specification (US): Single explicit constraints are removed, rendering the prompt strictly unable to communicate full task semantics.
• Syntax/Formatting (SF): Noise added without semantic shift, perturbing surface structure only.

Greedy decoding is enforced for determinism. All solutions are executed in true isolation, emphasizing robustness to both functional mis-specification and possible runtime errors.

Key Findings

Aggregate Robustness Is Structure-Dependent, Not Model-Dependent

Across all scales, models exhibit significant Pass@1 reductions on HumanEval under both US and LV mutations (−11.8% and −7.1% average, respectively) but near-zero impact on LiveCodeBench (−0.9% and −0.2%). This is independent of model size or family, including API-access reasoning models.

Sensitivity Patterns Still Reveal Deep Structural Dependence

Pass@1 deteriorates regardless of the edit distance of LV mutations on HumanEval, emphasizing that surface-form reliance dominates. In contrast, LiveCodeBench shows that prompt robustness follows from redundancy—different specification regions allow the model to compensate for missing information.

Figure 1: Pass@1 degradation under LV on HumanEval scales uniformly with the edit distance, supporting the claim that code LLMs trained on minimal-specification data depend on lexical surface forms for retrieval and fail to generalize.

Figure 2: Model attention is concentrated in the description region on HumanEval, but more evenly distributed (notably into Sample I/O and I/O format) on LiveCodeBench, supporting the mechanism underlying increased robustness under richer prompt structure.

Prompt Mutation Can Improve Correctness

Crucially, net-zero aggregate sensitivity on LiveCodeBench is shown to conceal two opposed effects: for each broken solution via mutation, there is a comparable number of correction flips (Fail→Pass), a symmetry not observable on HumanEval (Fail→Pass:Pass→Fail ≈ 0.4 for HE, ≈1.0 for LCB under LV).

Manual root cause analysis reveals that consistent improvements occur when:

• Retrieval cues (domain-specific vocabulary, overly precise or misleading identifiers, and explicit constraints) prime the model towards incorrect memorized solutions.
• Under-specification removes these, forcing genuine semantic reasoning over superficial retrieval and frequently activating more canonical and general algorithmic patterns.

Empirical Example: Removing an explicit numeric bound led the model, instead of enacting an incorrect shortcut, to apply a two-pass BFS—solving the problem as intended, where the bound had been acting as a misleading retrieval anchor.

Mechanisms Behind Improvements

Fine-grained analysis categorizes the drivers of improvement:

• Variable/Renaming Disruption (27% of LV-induced correct flips): Rewriting variables prevents retrieval of wrong memorized templates.
• Terminology Generalization (27%): Removing domain-specific language leads models to prefer robust, generic algorithms.
• Constraint Line Removal (39% of US improvements): Strips away anchors for LeetCode-style parsing or mistaken optimizations that are irrelevant or incorrect for the problem at hand.

These results challenge the notion that more information and increased specification necessarily improve model performance. Rather, prompt features that act as retrieval cues for the pretraining distribution can override problem-specific reasoning and introduce systematic failure modes. Specification completeness is thus revealed as a double-edged sword.

Theoretical and Practical Implications

For Practitioner Prompt Engineering

Care must be taken to avoid over-specification—detailed constraints and verbose context might not only fail to improve correctness but can degrade it by anchoring LLM solutions toward misapplied or shallow retrieval patterns. Empirically, in realistic multi-specification settings resembling LiveCodeBench, the recommendation is for lean and orthogonally redundant prompts rather than maximally complete single-layer prompts.

For Benchmark Design

Benchmark sensitivity studies performed solely on HumanEval-like datasets overestimate LLM fragility and do not generalize. Robust evaluation should move to contamination-free, structurally redundant benchmarks that reflect real-world programming scenarios and exploit natural redundancy.

For Model Pretraining and Alignment

Retrieval bias—a legacy of pretraining on code repositories with inconsistent or misleading comments and signatures—is not eliminated by scale or API-level fine-tuning. Approaches that emphasize generalization over memorization and mechanisms for cue-invariant reasoning (such as constraint-agnostic pretraining, controlled prompt normalization, or explicit retrieval-agnostic alignment) are called for.

Conclusion

The paper provides rigorous experimental evidence that prompt under-specification can, under practical and structurally rich prompting regimes, improve LLM code generation correctness. Prompt robustness is primarily a function of structure and redundancy, not model architecture. The findings demand new directions for benchmark design, LLM evaluation, and prompt engineering, emphasizing the need to counteract harmful retrieval cues and reconsider current notions of prompt completeness for code LLMs.

Reference: "When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation" (2604.24712).