Inferring Code Correctness from Specification

Abstract: LLMs have become integral to modern software development, enabling automated code generation at scale. However, validating the correctness of LLM-generated code remains a critical and largely unsolved challenge. Existing approaches either rely on dynamic consensus across multiple code candidates - making them costly and difficult to scale - or on static reasoning that is susceptible to dynamic bugs and order bias. In this paper, we propose TRAILS~ (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs. TRAILS~ first generates diverse test inputs via category partitioning based on the specification, then executes them against the candidate code and prompts LLMs to assess whether the resulting input-output pairs conform to the specification - without ever reasoning over the code itself. Scores are aggregated across inputs, to determines whether the program is likely correct. We evaluate TRAILS~ on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline. TRAILS~ improves Matthew Correlation Coefficient by up to 39\% relative to Zero-Shot COT and consistently outperforms HoarePrompt. Beyond accuracy, TRAILS~ demonstrates greater stability across seeded runs, reducing sensitivity to LLM non-determinism, and assigns correct labels to a larger set of unique code samples than competing approaches.

• The paper introduces TRAILS, a framework that decouples correctness validation from code reasoning by leveraging specification-grounded input-output pairs.
• By partitioning scenarios and applying input repair loops, TRAILS generates diverse, validated test inputs that expose dynamic bugs and logic inconsistencies.
• Empirical results show that TRAILS outperforms standard methods with up to a 39% MCC gain and improved label stability across code generation benchmarks.

Inferring Code Correctness from Specification: TRAILS

Problem Context and Motivation

Automated code generation using LLMs has transformed software engineering workflows, but code correctness validation remains a significant bottleneck. Reliance on LLM-generated test cases or static reasoning leads to compounding errors, test oracle deficiencies, and susceptibility to dynamic bugs and order biases. The paper introduces TRAILS (Targeted Reasoning Agreement via Inputs and Specifications), a framework that decouples correctness validation from code reasoning by leveraging concrete input-output pairs generated through categorical partitioning anchored in the specification.

Figure 1: Zero-shot reasoning versus TRAILS—TRAILS leverages specification-grounded input-output pairs to expose code logic contradictions missed by traditional approaches.

The motivation follows documented challenges including overtrust in LLMs by non-experts, improper validation, and the inability of LLMs to reliably generate aligned test suites in the absence of reliable oracles. Unlike consensus-based dynamic filtering—which scales poorly and requires multiple candidate codes—or logic-driven approaches such as HoarePrompt, which miss dynamic bugs or suffer decision instability, TRAILS anchors the validation process in specification-centric reasoning.

TRAILS Framework and Methodology

TRAILS operates as a two-phase input-grounded correctness classifier, focusing exclusively on reasoning over the specification, bypassing the code itself in the validation trace.

Figure 2: TRAILS schematic overview, detailing the stages of scenario extraction, input generation/repair, output collection, and specification-driven correctness inference.

1. Scenario Extraction: TRAILS prompts LLMs to partition behavioral scenarios from the natural language specification, using free-form text to encapsulate preconditions and abnormal cases.
2. Input Generation/Repair: For each partition, candidate inputs are synthesized and validated via code execution. Invalid inputs, identified by unhandled exceptions, invoke repair loops bounded by token budget constraints.
3. Deduplication and Coverage Reduction: Inputs are deduplicated using coverage heuristics, preserving code diversity and minimizing redundancy.
4. Output Collection: Each input is executed on the candidate code, yielding serialized outputs.
5. Specification-Centric Verification: LLMs are prompted to assess triplets (input, output, specification), assigning binary correctness labels with stepwise reasoning.
6. Aggregation and Decision Thresholding: Agreement rates across inputs constitute probabilistic correctness scores, subject to empirical or calibrated thresholds.

This pipeline treats specification as the sole ground truth, instantiating a low-trust system and enabling robust inference over single code samples without reliance on test oracle artifacts.

Empirical Evaluation

TRAILS is systematically evaluated on two benchmarks: LiveCodeBench (generated code) and CoCoClaNeL (competitive programming scenarios with human-written code), using three mid-sized open-source LLMs (Qwen3Coder-30B, Devstral-Small-24B, Olmo3.1-Instruct). Baselines include HoarePrompt (formal logic reasoning) and Zero-shot Chain-of-Thought (COT) prompting.

Key metrics:

• Matthews Correlation Coefficient (MCC): Quantifies classifier performance across true/false positive/negative rates, robust against imbalance.
• P4 metric: Symmetric extension of F1, sensitive to extreme confusion matrix cases.

Numerical Results:

• TRAILS yields up to 39% relative MCC gain and 20% P4 improvement compared to Zero-shot COT.
• Across all models and datasets, TRAILS demonstrates superior label stability under seeded runs, especially for incorrect code labels, minimizing false positives induced by LLM stochasticity.

The computational overhead mainly derives from input repair and generation, yet remains comparable to HoarePrompt and is tractable for consumer-grade hardware. Input diversity and repair mechanisms are particularly crucial for datasets requiring complex input formats (e.g., CoCoClaNeL’s stdin injection).

Comparative Analysis and Limitations

TRAILS outperforms baseline approaches for both generated and human-written code, exhibiting better robustness and lower sensitivity to model non-determinism. The empirical stability analysis confirms its reliability for task-level validation.

However, limitations include:

• Scenario Deficiency: Edge cases unrepresented in scenario extraction (e.g., alternative substring positions) can impede correctness inference.
• Execution-Centric Blind Spots: Bugs manifested only in unexercised paths (crashed partitions) are not verifiable, giving an advantage to purely static reasoning methods when executions fail prematurely.
• Token Costs: Despite optimization, repair phases incur higher token expenditures for malformed code, especially on tasks with atypical inputs.

Non-overlap in correctly labeled codes by TRAILS versus other methods suggests complementarity, motivating ensemble techniques for future work.

Theoretical and Practical Implications

The contribution is significant for specification-centered correctness inference without reliance on trusted oracles, test suites, or domain-specific metamorphic relations. TRAILS sustains applicability for practical large-scale code review, particularly for non-expert users, and provides a reliable means of mitigating LLM overtrust and validation gaps. Its design is extensible, enabling incorporation into both automated grading and real-world deployment scenarios.

Advancements in input scenario extraction and reparative prompt tuning (or dedicated fine-tuning of input synthesis) may further improve coverage and reduce costs. An ensemble combining TRAILS with formal logic and differential techniques can transcend current bottlenecks, addressing edge cases and execution-induced blind spots with greater generality.

Conclusion

TRAILS offers an authoritative methodology for inferring code correctness from specification using input-grounded LLM reasoning, outstripping state-of-the-art baselines in accuracy and decision stability. The framework addresses longstanding oracle deficits and validation challenges in LLM-driven code workflows. Future directions include improving scenario coverage, optimizing token costs, and exploring ensemble verification strategies integrating diverse approaches. TRAILS sets a foundation for reliable, scalable correctness inference in automated code generation ecosystems.