DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode

Abstract: This work addresses test output prediction, a key challenge in test case generation. To improve the reliability of predicted outputs by LLMs, prior approaches generate code first to ground predictions. One grounding strategy is direct execution of generated code, but even minor errors can cause failures. To address this, we introduce LLM-based pseudocode execution, which grounds prediction on more error-resilient pseudocode and simulates execution via LLM reasoning. We further propose DuET, a dual-execution framework that combines both approaches by functional majority voting. Our analysis shows the two approaches are complementary in overcoming the limitations of direct execution suffering from code errors, and pseudocode reasoning from hallucination. On LiveCodeBench, DuET achieves the state-of-the-art performance, improving Pass@1 by 13.6 pp.

• The paper introduces DuET, a dual-path framework that combines deterministic execution of generated code with LLM-simulated execution of pseudocode using path-weighted functional majority voting.
• DuET achieves 81.1 Pass@1 with GPT-4-Turbo on LiveCodeBench, improving on prior grounded methods while remaining strongest on the contamination-free January–April 2024 subset.
• The method improves CodeT-style candidate filtering by 3.2 percentage points on LiveCodeBench-Easy and works because code execution and pseudocode simulation fail in complementary ways across implementation errors and reasoning hallucinations.

Motivation and problem statement

Test case generation decomposes into input generation and output prediction, and the paper focuses on the latter: given a problem description $d$ and a test input $i$, predict the corresponding output. This task requires precise program reasoning and remains difficult even for strong LLMs (2604.11514). The dominant grounded approach, TestChain, generates code from the problem description and executes it directly, so prediction correctness is entangled with implementation quality: models often capture the correct algorithmic logic but mistranslate it into executable code due to minor implementation errors such as off-by-one bugs or missing edge cases.

The paper's central observation is that direct execution and LLM-based reasoning fail under complementary conditions. Direct execution is deterministic but brittle to implementation errors; LLM-based simulation avoids those errors but is prone to execution hallucinations, where plausible intermediate reasoning steps diverge from the intended logic, particularly for long or deeply nested control flow. DuET exploits this complementarity rather than choosing one path.

Method

DuET first disentangles code generation into pseudocode generation followed by translation, $g = t \circ p$, and grounds prediction on the pseudocode $p(d)$ directly, bypassing the translation step $t$. An LLM then simulates execution of the pseudocode to produce outputs — the paper's first use of LLM-based pseudocode execution for test output prediction. This path is decoupled from code correctness, so implementation errors in generated code cannot corrupt predictions; the cost is exposure to hallucinated multi-step reasoning.

The full framework runs two paths in parallel: (1) direct code execution of translated code, and (2) LLM-based pseudocode execution. Outputs are aggregated via functional majority voting (FMV), extended with a path-weighted variant: if all valid outputs within a path unanimously agree, that path's votes receive a binary weight boost ($w_{high} > w_{base}$). This weighting adds no inference cost and yields consistent small gains without degradation. The authors set equal sample budgets across paths ($l = m$) for simplicity, though the framework permits asymmetric allocation.

Results on test output prediction

On LiveCodeBench's test output prediction benchmark (May 2023–April 2024 split), DuET with GPT-4-Turbo-2024-04-09 as backbone achieves a state-of-the-art Pass@$1$ of 81.1, exceeding GPT-4-Turbo with FMV by 6.9 pp and TestChain with FMV by 5.6 pp. On the contamination-free January–April 2024 subset, DuET attains the highest Pass@$1$ among grounded methods across all evaluated LLM backbones. The abstract reports an overall improvement of 13.6 pp in Pass@$1$ over prior approaches.

Ablations isolate each design choice. Direct code execution outperforms LLM-based code execution (75.5 vs. 72.2 Pass@$i$0), confirming that faithful runtime behavior beats simulated execution when both are grounded on code. Conversely, pseudocode is a better grounding than code for LLM-based execution (74.1 vs. 72.2). Combining direct code execution with LLM-based pseudocode execution — DuET's configuration — reaches 80.8, supporting the claim that the two paths are best combined rather than substituted.

Subset analyses sharpen this picture. When generated code passes all test inputs, direct execution achieves 100% accuracy while LLM-based pseudocode execution reaches 86.7%, limited by hallucinations; when generated code fails at least one input, direct execution collapses to 39.4% while pseudocode execution holds at 53.9%. Grouping by execution trace length shows LLM-based pseudocode execution outperforming direct execution up to roughly 100 trace steps, after which its accuracy degrades while direct execution remains reliable. These results imply that neither path dominates uniformly, and that DuET's gains come precisely from instances where the two failure modes are anti-correlated.

Impact on end-to-end code generation

Integrating DuET into CodeT-style candidate filtering improves Llama-3.1-8B-Instruct's Pass@$i$1 on LiveCodeBench-Easy by 3.2 pp over CodeT. A notable negative result: replacing it with TestChain hurts performance by 5.6 pp relative to CodeT. The authors attribute this to a zero-advantage problem analogous to zero-variance prompts in GRPO-style RL: inferring test outputs from candidate programs and reusing them to rank the same pool provides little signal when most candidates are correct and unreliable signals when most are wrong, so filtering is least informative exactly when it is most needed. Because LLM-based pseudocode execution is decoupled from the candidates being ranked, DuET sidesteps this circular dependency.

Generalization checks reinforce the main claims. With the reasoning model QwQ-32B on BigCodeBench-Hard, DuET is the only prediction method that outperforms no filtering at all, achieving 25.3 Pass@$i$2, indicating robustness to outputs involving arbitrary external library objects. On repo-level DevEval, LLM-based pseudocode execution dominates in domains where baseline generation quality is low, direct execution dominates in easier domains, and DuET ranks first or second across all domains. The paper also reports best filtering results on HumanEval(+) and DevEval alongside LiveCodeBench-Easy and BigCodeBench-Hard.

Limitations and open questions

The paper concedes several points. The relative reliability of the two paths varies with trace length, suggesting — but not yet realizing — adaptive per-instance weighting within FMV; the current scheme uses static path weights. Test input generation is intentionally simple, and more diagnostic inputs such as edge cases might further improve downstream filtering. Evaluation of newer models was constrained by the LiveCodeBench setup, which had not been updated for the test output prediction task at the time. Additionally, the main leaderboard includes models whose knowledge cutoffs postdate the benchmark window, so some reported comparisons carry contamination risk, which the authors partially mitigate via the later cutoff-free subset. Whether adaptive confidence estimation can outperform the fixed binary boost, and how the method scales to very long traces where both paths degrade, remain open.

Conclusion

DuET frames test output prediction as a choice between two failure modes — implementation errors in generated code versus execution hallucinations in LLM reasoning — and resolves it by running both a direct code execution path and an LLM-based pseudocode execution path, aggregated through path-weighted functional majority voting. It sets a new state of the art on LiveCodeBench test output prediction, transfers to end-to-end code generation via candidate filtering across four benchmarks, and identifies the zero-advantage problem that makes pure execution-based prediction counterproductive in filtering pipelines.