Generate–Execute–Refine Loop

• Generate–Execute–Refine Loop is an iterative computational pattern that produces candidate artifacts, executes them to gather feedback, and refines outcomes using localized corrections.
• It applies structured intermediate representations and targeted execution signals in domains like code synthesis, test generation, and proof-of-concept evaluations.
• Empirical findings reveal enhanced performance through precise feedback loops, though challenges remain in managing regression and broad search spaces.

Generate–Execute–Refine (GER) denotes an iterative computational pattern in which a system first produces a candidate artifact, then runs that artifact against an external environment, evaluator, or execution substrate, and finally updates either the artifact itself or the mechanism that produced it using the resulting feedback. In the recent literature, the generated artifact ranges from synthetic evaluation cases, loop-verification tasks, code, tests, SQL stage outputs, Python forensic tool chains, network emulation scenarios, and trace-derived program fragments; execution ranges from LLM judging, proof search, program execution, mutation testing, database execution, image-tool runtime, and network emulation; refinement may target the artifact, a search policy, a curriculum generator, stage-specific prompts, or an intermediate representation (Do et al., 6 Nov 2025, Laurent et al., 2022, Wang et al., 2024, Zhang et al., 18 Dec 2025, Mohr et al., 10 Jan 2026, Ravi et al., 24 Mar 2026, Guo et al., 2023, Adam et al., 2 Jun 2026).

1. Canonical decomposition

Across the cited works, GER is not a single algorithm but a recurring control structure. The generate phase proposes candidates under explicit controls or latent policies. The execute phase subjects those candidates to a structured environment that returns observable signals. The refine phase incorporates those signals into a revised candidate, a revised policy, or a revised generator. This pattern is explicit in synthetic-data-assisted LLM judging, where users configure task type, domain, persona, length, and borderline cases before running an LLM judge and then revising the rubric; in loop invariant synthesis, where a teacher generates verification tasks and a solver executes nondeterministic proof search; and in code-and-test systems, where candidate programs and tests are repeatedly run and filtered through execution outcomes (Do et al., 6 Nov 2025, Laurent et al., 2022, Wang et al., 2024).

The same decomposition appears in agentic settings. ForenAgent generates Python tool calls or compositions for image forgery detection, executes them in a sandbox, and refines its next action through reflective reasoning. Reflect-SQL generates typed stage outputs for text-to-SQL, executes the realized SQL under syntactic and semantic judges, and refines the implicated generation stage rather than rewriting the entire query. AgentCyTE generates CORE XML threat scenarios, executes them in a deterministic network emulator, and refines them through structured validation and hypothesis-driven repair (Zhang et al., 18 Dec 2025, Mohr et al., 10 Jan 2026, Rodriguez et al., 29 Oct 2025).

A distinct but related form appears in execution-centered programming systems. “A Systematic Approach to Programming” begins from manually produced execution traces and progressively derives operations, control flow, and predicates from them. AlgoTouch similarly records direct data manipulations as primitive operations, executes them immediately on an explicit notional machine, and refines partial conditionals or loop macros as new executions expose missing branches or termination conditions. In these cases, GER is embedded in the act of program construction itself rather than added after an initial artifact exists (Chandoo, 2018, Adam et al., 2 Jun 2026).

2. Representations and formal mechanisms

A common property of GER systems is the use of structured intermediate representations that make refinement localizable. Reflect-SQL factorizes text-to-SQL as $$\mu \coloneqq \pi_T \circ \pi_{T-1} \circ \cdots \circ \pi_1$$ over typed stages such as schema understanding, value grounding, aggregation and predicate alignment, and SQL realization, while tracking validated constraints $C_{\text{validated}}(t) \subseteq C$ so that later updates preserve earlier correct decisions (Mohr et al., 10 Jan 2026). PER-KBQA represents reasoning as KoPL programs serialized with <func> and <arg>, executes each step over the knowledge base, and appends <return> traces to form an execution context for answer generation (Guo et al., 2023). AlgoTouch represents execution traces as

$E = (s_0, o_1, s_1, \dots, o_n, s_n),$

and packages iterative behavior into AGT macros with From, Until, Loop, and Terminate blocks (Adam et al., 2 Jun 2026).

In automated reasoning, GER is expressed more formally as interacting nondeterministic programs. The loop-invariant synthesis system models both teacher and solver with choose and event operators, treats each choose point as an MDP state, and refines both sides by AlphaZero-style training with event-aware value prediction. The solver searches for invariants satisfying initiation, preservation, and post-condition obligations, while the teacher searches over valid generated problems and soft constraint satisfaction (Laurent et al., 2022).

Code-and-test systems define equally explicit execution objects. GenX maintains separate sequence-to-sequence models for code and test generation and constructs a binary pass/fail matrix $P \in \{0,1\}^{C \times T}$ at inference time, iteratively updating code and test scores by

$$s_{\text{code}}^{(k+1)} = \frac{P \cdot s_{\text{test}}^{(k)}}{\mathbf{1}^T s_{\text{test}}^{(k)} + 10^{-8}}, \qquad s_{\text{test}}^{(k+1)} = \frac{P^T \cdot s_{\text{code}}^{(k+1)}}{\mathbf{1}^T s_{\text{code}}^{(k+1)} + 10^{-8}}.$$

LLMLOOP instead organizes refinement around a concrete Java/Maven project representation with source files, dependencies, tests, PMD reports, and PIT mutants, so that each loop can target a specific failure channel (Wang et al., 2024, Ravi et al., 24 Mar 2026).

3. Execution channels and feedback signals

Execution in GER is valuable only insofar as it yields actionable feedback. Several systems therefore emphasize observability rather than mere verdicts. The LLM-judge refinement tool exposes not only the judge’s categorical decision and rationale but also the exact generation and judging prompts through a “View explanation” affordance. The same interface stores generation prompts for each synthetic test case, including borderline cases synthesized from the current criterion definitions, so users can diagnose whether disagreement originates in the rubric, the prompt, or the case itself (Do et al., 6 Nov 2025).

Other systems operationalize feedback through formal or runtime diagnostics. In loop-invariant synthesis, failures arise from proof obligations such as failed preservation, and the system records event penalties for abduction and conjecturing while using MCTS-guided trajectories to update solver and teacher policies. In PER-KBQA, execution returns intermediate entities or literals at each program step; refinement then conditions on those traces rather than on the raw logical form alone (Laurent et al., 2022, Guo et al., 2023).

Code-oriented GER work uses the full spectrum of software-engineering feedback. The iterative code-correction study employs compiler errors, runtime errors, failing test cases, expected-vs-actual outputs, and time or memory limit messages, and measures success with ISR@k and MIS; it reports that Wrong Answer and TLE account for about 95% of all failures, while Compile Error and Runtime Error are much more tractable (Zhang et al., 16 Jun 2026). OpenCodeInterpreter formalizes multi-turn execution feedback as execution outputs, exceptions, and optional synthetic human guidance inside dialogue state, and trains on 68K multi-turn Code-Feedback interactions (Zheng et al., 2024). LLMLOOP extends the feedback surface further by adding PMD static-analysis violations, JUnit failures, and PIT surviving mutants, each routed into its own refinement loop (Ravi et al., 24 Mar 2026).

Agentic execution environments produce richer multimodal signals. ForenAgent receives heatmaps, spectra, scores, thumbnails, and runtime errors from a whitelisted Python toolbox, while AgentCyTE receives XSD validation errors, connectivity matrices, routing inconsistencies, segmentation violations, and service-state observations from CORE-based network emulation (Zhang et al., 18 Dec 2025, Rodriguez et al., 29 Oct 2025). This suggests that GER becomes more effective when execution returns structured diagnostics rather than a single scalar success flag.

4. Localization, reflection, and human oversight

A major line of development within GER is the move from whole-artifact rewriting to localized refinement. Reflect-SQL is explicit on this point: violations are localized to the responsible stage, and the update is applied to the stage parameterization $\theta_t$ rather than to the current SQL string. The stated goal is to maximize preservation of previously validated constraints and support monotonic improvement over a query set (Mohr et al., 10 Jan 2026). REFINE adopts the same logic for educational feedback: its judge produces binary component-level judgments and targeted explanations, and regeneration rewrites only the failing feedback components rather than the entire response (Fawzi et al., 31 Mar 2026).

Human-in-the-loop systems make localization concrete in the interface. In the LLM-as-a-judge tool, users inspect disagreements between expected and judged outcomes, revise criterion descriptions, generate targeted new cases, and use inline span-level operations—Paraphrase, Elaborate, Shorten, Regenerate—to perturb only the highlighted region while preserving surrounding context (Do et al., 6 Nov 2025). AlgoTouch does something analogous for imperative programs: a conditional may be created with an else branch left as TODO, and that branch is only completed when a later concrete execution reaches it; loops are assembled non-linearly by first recording a representative body, then termination conditions, then initialization (Adam et al., 2 Jun 2026).

Reflective behavior is especially prominent in agentic tool-use systems. ForenAgent’s loop includes global perception, local focusing, iterative probing, and holistic adjudication, with reflection triggered by incoherent cues or tool failure. The paper describes recovery actions such as recropping to an alternate ROI, switching tool families, or adjusting JPEG quality grids, all within a hard tool budget of $T_{\max}=7$ (Zhang et al., 18 Dec 2025). AgentCyTE similarly uses a hypothesis agent to generate multiple causal explanations of failure and multiple repair candidates, then selects the variant minimizing residual errors, thereby avoiding deterministic repair cycles (Rodriguez et al., 29 Oct 2025).

These designs collectively shift GER from simple retry logic toward controlled revision. A plausible implication is that refinement quality depends less on the number of iterations than on whether the system can identify which subcomponent failed and preserve what is already correct.

5. Reported empirical outcomes

The empirical record is heterogeneous, but several papers report measurable gains from GER-style execution feedback or localized refinement.

Setting	Reported outcome	Reference
Synthetic data for LLM judges	83% preferred the tool; synthetic cases: $M=9.08$ vs $5.21$; final alignment $0.57$ vs $C_{\text{validated}}(t) \subseteq C$0 and not significant	(Do et al., 6 Nov 2025)
Loop invariant synthesis	Greedy solve rate on Code2Inv: random $C_{\text{validated}}(t) \subseteq C$1, untrained teacher $C_{\text{validated}}(t) \subseteq C$2, full teacher $C_{\text{validated}}(t) \subseteq C$3; with MCTS the nondeterministic strategy solves 100%	(Laurent et al., 2022)
GenX on APPS+	Code pass@1 for Intro/Inter/Comp: $C_{\text{validated}}(t) \subseteq C$4; ranked $C_{\text{validated}}(t) \subseteq C$5: $C_{\text{validated}}(t) \subseteq C$6; ranked $C_{\text{validated}}(t) \subseteq C$7: $C_{\text{validated}}(t) \subseteq C$8	(Wang et al., 2024)
ForenAgent on FABench	Accuracy $C_{\text{validated}}(t) \subseteq C$9, F1 $E = (s_0, o_1, s_1, \dots, o_n, s_n),$0; removing Cold Start gives $E = (s_0, o_1, s_1, \dots, o_n, s_n),$1 and removing RFT gives $E = (s_0, o_1, s_1, \dots, o_n, s_n),$2	(Zhang et al., 18 Dec 2025)
Reflect-SQL	Spider: $E = (s_0, o_1, s_1, \dots, o_n, s_n),$3 EX; BIRD: $E = (s_0, o_1, s_1, \dots, o_n, s_n),$4 EX and $E = (s_0, o_1, s_1, \dots, o_n, s_n),$5 VES; GPT-5 variant: Spider $E = (s_0, o_1, s_1, \dots, o_n, s_n),$6 EX, BIRD $E = (s_0, o_1, s_1, \dots, o_n, s_n),$7 EX and $E = (s_0, o_1, s_1, \dots, o_n, s_n),$8 VES	(Mohr et al., 10 Jan 2026)
LLMLOOP on HumanEval-X	pass@1 rises from $E = (s_0, o_1, s_1, \dots, o_n, s_n),$9 to $P \in \{0,1\}^{C \times T}$0; pass@10 rises from $P \in \{0,1\}^{C \times T}$1 to $P \in \{0,1\}^{C \times T}$2	(Ravi et al., 24 Mar 2026)

The gains are not uniform in mechanism. The iterative code-correction study finds that reasoning models improve steadily across iterations, whereas non-reasoning models often plateau after one or two iterations; Compile Error is fixed about $P \in \{0,1\}^{C \times T}$3 of the time and Runtime Error about $P \in \{0,1\}^{C \times T}$4, but Wrong Answer is fixed only about $P \in \{0,1\}^{C \times T}$5 and TLE about $P \in \{0,1\}^{C \times T}$6 (Zhang et al., 16 Jun 2026). “Feedback Over Form” reaches a closely related conclusion at 1–3B scale: self-refinement with execution feedback improves code generation by more than 4 standard deviations on both HumanEval and MBPP, but the mechanism is narrow, fixing many NameError and SyntaxError cases while rarely fixing AssertionError, and every forced iteration is net-negative without early stopping (McAndrews, 23 Apr 2026).

The code-agent literature reports similar benefits from multi-turn execution traces. OpenCodeInterpreter-33B reaches an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, rising to 91.6 (84.6) with synthesized human feedback from GPT-4 (Zheng et al., 2024). In educational feedback, REFINE reports that judge-guided regeneration significantly improves feedback quality, and that the fine-tuned 8B interactive agent is not significantly different from GPT-5 on relevance, actionability, or tool relevance; its mean steps are 2.26 versus 2.39 for the 8B Thinking baseline (Fawzi et al., 31 Mar 2026).

Taken together, these results indicate that GER is most consistently beneficial when the execution layer provides precise, localizable signals and when refinement is constrained enough to preserve earlier correct structure.

6. Limitations, failure modes, and design tensions

The literature also records recurring limitations. Synthetic-data GER can plateau in diversity: the LLM-judge study notes repetition under the same configuration despite high temperature, “clean” outputs that lack real-world messiness, and practitioner concerns about circularity or model collapse when model-generated data is used for both generation and judging (Do et al., 6 Nov 2025). GenX reduces false positives by executing generated tests on ground-truth code and filtering code against accumulated tests, but its authors still identify diversity and representativeness as open questions, and pass@10 can drop relative to APPS even as pass@1 improves on APPS+ (Wang et al., 2024).

In search-based reasoning systems, refinement can be compute-intensive or theory-limited. Loop-invariant synthesis still relies on MCTS during training, and its current abduction engine is restricted to linear integer arithmetic; the authors explicitly note brittleness on out-of-distribution programs and the engineering effort required to design useful teacher constraints (Laurent et al., 2022). Reflect-SQL identifies non-convergence within budget, critic mislocalization, lock-in to incorrect beliefs, and incomplete semantic-checker coverage as failure modes, even though its stage-localized preservation strategy mitigates some drift (Mohr et al., 10 Jan 2026).

Code GER highlights a separate tension between iteration and regression. The iterative code-correction study shows that wrong answers and TLE dominate the failure distribution and are harder to fix because they require algorithmic redesign rather than local patching (Zhang et al., 16 Jun 2026). “Feedback Over Form” makes the regression issue explicit: without early stopping, every iteration is net-negative, because refiners frequently damage already-correct code (McAndrews, 23 Apr 2026). LLMLOOP adds additional validation channels, but at the cost of repeated LLM invocations and tool runs, and the paper explicitly notes that runs can take several minutes (Ravi et al., 24 Mar 2026).

Execution-centered programming work exposes a different boundary. “A Systematic Approach to Programming” requires careful selection of traces until the control-flow graph and guards stabilize, and the underlying synthesis problem is formalized through restricted graph coloring rather than solved automatically in full generality (Chandoo, 2018). AlgoTouch reduces search by making the user choose guards and delimit loop structure from concrete executions, but ambiguous guard selection, nondeterministic operations, and richer data structures remain out of scope (Adam et al., 2 Jun 2026).

Across domains, the same tension recurs: GER is strongest when execution feedback is informative, inexpensive, and localizable, but weaker when the feedback is sparse, the search space is too large, or refinement rewrites more than it preserves.