Execution-Based Code Generation PBS

• Code Generation and Execution-based PBS is a framework that integrates runtime execution feedback into code synthesis to ensure correctness and usability.
• It leverages reinforcement learning, iterative co-training, and execution-guided decoding to align generated code with real-world execution outcomes.
• Robust evaluation metrics like pass@k and comprehensive benchmarking validate improvements in compilation success, resource efficiency, and model transferability.

Code Generation and Execution-based PBS

Code generation and execution-based Program Behavior Scoring (PBS) frameworks systematically integrate program execution feedback into the process of synthesizing, optimizing, and evaluating code produced by both conventional and learning-based models. In these approaches, the design of model objectives, learning algorithms, and evaluation metrics is informed by the actual behavior of generated programs during compilation, interpretation, and testing. Execution-based PBS methods are critical for ensuring syntactic correctness, semantic equivalence, and practical usability of machine-generated code across diverse domains and languages.

1. Formal Definitions and Problem Setting

Execution-based code generation models reformulate code synthesis as a sequence modeling problem governed by program execution outcomes. Rather than optimizing purely for surface-level token accuracy, modern systems utilize multi-stage protocols that involve runtime execution or simulation, verification against specification tests, and alignment with intermediate or final program semantics.

Markov Decision Process Formulation: Methods such as PPOCoder formalize code generation as a finite-horizon Markov decision process (MDP) where, at each timestep $t$, the agent’s state $s_t = (\hat{y}_{<t}, x)$ consists of the current prompt and the sequence of previously output code tokens. Actions are next-token emissions $a_t = \hat{y}_t$, and policies $\pi_\theta$ are parameterized by pretrained or finetuned LLMs (Shojaee et al., 2023).

Reinforcement Learning Objectives: Sequence-level rewards leverage non-differentiable signals from compilation results, execution pass/fail traces, structural code alignment (e.g., AST/DFG matches), or semantic outcomes. This facilitates the incorporation of external code-specific knowledge directly into model optimization, transcending limitations of standard cross-entropy based supervised learning.

2. Algorithmic Frameworks Incorporating Execution Feedback

Reinforcement Learning with Execution Feedback

Execution-based RL frameworks employ actor-critic architectures—such as Proximal Policy Optimization (PPO) and Group-Relative Policy Optimization (GRPO)—to align LLM outputs with program-level correctness (Shojaee et al., 2023, Jiang et al., 21 Oct 2025). Key system components include:

• Value Networks: Predict expected returns from partially generated programs, often sharing Transformer backbone parameters with the policy network.
• Reward Design: Multi-factor reward signals blend compilation success, test outcomes, syntactic/semantic structure match, and exploration-penalizing KL divergences. For example, in PPOCoder:

$r_t = 𝟙(\hat{y}_t = \langle\text{eot}\rangle)[R_{cs} + R_{ast} + R_{dfg} - \beta R_{kl}(s_t)] + 𝟙(\hat{y}_t \neq \langle\text{eot}\rangle)[-\beta R_{kl}(s_t)]$

• Execution Semantics Alignment: Recent advances (e.g., CodeRL+) extend standard RL with auxiliary objectives that require the model to infer variable-level execution trajectories on failed rollout traces, integrating a dense reward signal for accurately predicting post-execution variable values:

$R_{\mathrm{sem}}^{(i)} = \frac{1}{|V|}\sum_{k=1}^n 𝟙 [\hat{v}_k^{\mathrm{final}} = v_k^{\mathrm{final},*}]$

(Jiang et al., 21 Oct 2025).

Iterative Co-training of Code and Test Generators

GenX introduces a bootstrapped co-training loop where code and test generators alternate in expanding the set of solutions and test cases. Test augmentation uses a gold solution to label new test inputs via execution, while code augmentation admits only generated code that passes all available tests. A dual-critic iterative scoring function is then used to mutually reinforce rankings of code and test candidates (Wang et al., 2024).

Execution-Guided Decoding and Self-Debugging

Frameworks such as DOCE and EG-CFG exploit execution signals at inference by:

• Filtering and Reranking: Candidates are filtered based on pass/fail traces from (potentially small) sets of trial unit tests. Minimum Bayes Risk decoding is performed according to execution-based utility, which compares test output vectors for majority agreement (Li et al., 2024).
• Self-Debugging: Multi-candidate debugging uses execution feedback to prompt the LLM to repair failed solutions; the resulting population is then re-ranked using additional execution-based filtering and reranking steps.
• Line-by-Line, Real-Time Execution Guidance: EG-CFG interleaves token-level code generation with dynamic execution feedback via classifier-free guidance. At each line boundary, execution traces of beam search candidates are injected into the model prompt, guiding subsequent code emission toward, e.g., syntactic coherence and passing all tests (Lavon et al., 12 Jun 2025).

3. Execution-based Evaluation and Benchmarking

Core Execution-based Metrics

Functional correctness is evaluated using direct execution of generated code against curated unit test suites. The standard pass@k metric quantifies the fraction of prompts for which at least one of the top k generated variants passes all test cases:

$$\text{pass}@k = 1 - \frac{\binom{N-c}{k}}{\binom{N}{k}}$$

where $N$ is the number of generated samples and $c$ is the count of correct (fully passing) samples (Wang et al., 2022, Xie et al., 2024).

Benchmark Construction

Benchmarks such as Exec-CSN (Xie et al., 2024), CodeHaluEval (Tian et al., 2024), and ODEX (Wang et al., 2022) structurally couple code generation tasks with explicit runtime evaluation frameworks. Pipelines encompass:

• LLM-driven sandboxing and test generation: LLMs are prompted to create isolated evaluation snippets with test harnesses.
• Iterative execution-debug-filter cycles: Repeated execution against generated or human-written tests, with failures triggering sample exclusion or further repair.
• Taxonomic Error Analysis: Code hallucinations are classified by runtime failure type (e.g., mapping, naming, resource, logic), enabling detailed model diagnosis and targeted improvement (Tian et al., 2024).

Human-in-the-Loop and Automated Preference Arenas

Large-scale human preference platforms (e.g., BigCodeArena) log pairwise judgments of executed code in diverse domains. These judgments are compared to LLM-based reward models and Elo-tracked in automated evaluation benchmarks (e.g., AutoCodeArena), with execution artifacts found to strongly boost preference-model alignment and rating stability (Zhuo et al., 9 Oct 2025).

4. Systematic Design of Execution-based PBS Pipelines

Central PBS pipelines typically combine the following stages:

1. Prompting: Curate or synthesize natural language or formal specification prompts.
2. Candidate Generation: Sample diverse code variants using autoregressive LLM decoding (greedy/nucleus/high-temperature sampling).
3. Execution Filtering: Sanitize and sandbox raw generations, run against static or dynamically-augmented test suites, and exclude candidates with fatal execution errors.
4. Reranking and MBR Decoding: Apply execution-based scoring/reranking (dual-critic, filtering, utility-based) to select the best candidate(s).
5. Iterative Repair: Feed failed executions back into prompt contexts for further model-based refinement or self-debugging.
6. Evaluation: Quantify outcomes with pass@k, coverage, efficiency, and error-type frequencies.

The architecture is highly modular and applicable across languages, problem domains, and evaluation environments—requiring minimal per-language adaptation beyond interpreter and test harness availability (Shojaee et al., 2023, Xie et al., 2024, Wang et al., 2024).

5. Impact on Model Design and Empirical Advances

Generalization and Transfer

Execution-based PBS enables task- and language-agnostic adaptation of pre-trained code LLMs. Notable findings include:

• Massive increases in compilation and test pass rates when fine-tuning strong code models (e.g., CodeT5), with gains of up to +45.5 points on Python CodeSearchNet; robust cross-lingual and cross-task transfer to code completion, translation, and synthesis (Shojaee et al., 2023).
• Distributed and scalable annotation via LLMs leads to datasets covering hundreds of libraries and thousands of real-world GitHub functions, establishing challenging new benchmarks (e.g., Exec-CSN) (Xie et al., 2024).

Failure Analysis and Robustness

Systematic execution-based hallucination detection (e.g., CodeHalu) reveals persistent challenges in logic deviation, resource constraint management, and identifier handling, with leading LLMs still producing logic hallucinations in up to 85.8% of relevant cases. This motivates the integration of explicit execution feedback and modular defense strategies into future code generation systems (Tian et al., 2024).

Model Efficiency and Optimization

Frameworks such as FasterPy push execution-based PBS beyond correctness, targeting runtime efficiency by integrating retrieval-augmented generation over large performance-annotated codebases, yielding both significant Pass@1 and speedup improvements over standard LLM baselines (Wu et al., 28 Dec 2025).

Pipeline-level Latency Reduction

Parallel buffered scheduling and pipelined execution architectures (e.g., Eager) substantially reduce end-to-end code generation latency by overlapping token emission, chunk-level code execution, and chunk detection, with empirically demonstrated reductions in non-overlapped execution time of up to 99.9% (Sun et al., 1 Apr 2026).

6. Limitations and Future Directions

Execution-based PBS approaches, while powerful, face several limitations:

• Oracle Test Suite Coverage: Many methods rely on the thoroughness of unit tests for correctness attribution; rare edge cases or underspecified behaviors may remain undetected (Wang et al., 2022).
• Sandboxing and Security: Running arbitrary generated code poses risks; scalable, language-general, and reproducible sandboxing infrastructure, especially for scripting and systems code, is an active engineering challenge (Vo et al., 2024).
• Efficiency and Overhead: High-throughput code execution is costly compared to static metrics; approximations via LLM-predicted function metrics (e.g., CodeScore) or symbolic/execution hybrid analysis are being explored (Dong et al., 2023).
• Cross-Domain and Cross-Language Portability: Extensions to new programming languages, multi-runtime compatibility, and real-world complex code remain open technical fronts (Zheng et al., 2024, Jacobs et al., 2016).

Key open questions include automated generation of stronger tests, efficient transfer to domains without pre-existing test cases, and end-to-end learning integrating code generation, execution, and reranking. The consensus in recent literature is that future progress in code generation cannot disentangle modeling from execution-based alignment and evaluation, and that flexible, modular PBS systems will be central to advancing the robustness and utility of machine-generated code.