---
title: Generate–Execute–Refine Loop
url: https://www.emergentmind.com/topics/generate-execute-refine-loop
type: topic
---

# Generate–Execute–Refine Loop

Generate–Execute–Refine (GER) denotes an iterative computational pattern in which a system first produces a candidate artifact, then runs that artifact against an external environment, evaluator, or execution substrate, and finally updates either the artifact itself or the mechanism that produced it using the resulting feedback. In the recent literature, the generated artifact ranges from synthetic evaluation cases, loop-verification tasks, code, tests, SQL stage outputs, Python forensic tool chains, network emulation scenarios, and trace-derived program fragments; execution ranges from LLM judging, proof search, program execution, mutation testing, database execution, image-tool runtime, and network emulation; refinement may target the artifact, a search policy, a curriculum generator, stage-specific prompts, or an intermediate representation [2511.04478] [2205.14229] [2412.13464] [2512.16300] [2601.06678] [2603.23613] [2305.03356] [2606.03349].

## 1. Canonical decomposition

Across the cited works, GER is not a single algorithm but a recurring control structure. The generate phase proposes candidates under explicit controls or latent policies. The execute phase subjects those candidates to a structured environment that returns observable signals. The refine phase incorporates those signals into a revised candidate, a revised policy, or a revised generator. This pattern is explicit in synthetic-data-assisted LLM judging, where users configure task type, domain, persona, length, and borderline cases before running an LLM judge and then revising the rubric; in loop invariant synthesis, where a teacher generates verification tasks and a solver executes nondeterministic proof search; and in code-and-test systems, where candidate programs and tests are repeatedly run and filtered through execution outcomes [2511.04478] [2205.14229] [2412.13464].

The same decomposition appears in agentic settings. ForenAgent generates Python tool calls or compositions for image forgery detection, executes them in a sandbox, and refines its next action through reflective reasoning. Reflect-SQL generates typed stage outputs for text-to-SQL, executes the realized SQL under syntactic and semantic judges, and refines the implicated generation stage rather than rewriting the entire query. AgentCyTE generates CORE XML threat scenarios, executes them in a deterministic network emulator, and refines them through structured validation and hypothesis-driven repair [2512.16300] [2601.06678] [2510.25189].

A distinct but related form appears in execution-centered programming systems. “A Systematic Approach to Programming” begins from manually produced execution traces and progressively derives operations, control flow, and predicates from them. AlgoTouch similarly records direct data manipulations as primitive operations, executes them immediately on an explicit notional machine, and refines partial conditionals or loop macros as new executions expose missing branches or termination conditions. In these cases, GER is embedded in the act of program construction itself rather than added after an initial artifact exists [1808.08989] [2606.03349].

## 2. Representations and formal mechanisms

A common property of GER systems is the use of structured intermediate representations that make refinement localizable. Reflect-SQL factorizes text-to-SQL as $\mu \coloneqq \pi_T \circ \pi_{T-1} \circ \cdots \circ \pi_1$ over typed stages such as schema understanding, value grounding, aggregation and predicate alignment, and SQL realization, while tracking validated constraints $C_{\text{validated}}(t) \subseteq C$ so that later updates preserve earlier correct decisions [2601.06678]. PER-KBQA represents reasoning as KoPL programs serialized with `<func>` and `<arg>`, executes each step over the knowledge base, and appends `<return>` traces to form an execution context for answer generation [2305.03356]. AlgoTouch represents execution traces as
$$
E = (s_0, o_1, s_1, \dots, o_n, s_n),
$$
and packages iterative behavior into AGT macros with `From`, `Until`, `Loop`, and `Terminate` blocks [2606.03349].

In automated reasoning, GER is expressed more formally as interacting nondeterministic programs. The loop-invariant synthesis system models both teacher and solver with `choose` and `event` operators, treats each `choose` point as an MDP state, and refines both sides by AlphaZero-style training with event-aware value prediction. The solver searches for invariants satisfying initiation, preservation, and post-condition obligations, while the teacher searches over valid generated problems and soft constraint satisfaction [2205.14229].

Code-and-test systems define equally explicit execution objects. GenX maintains separate sequence-to-sequence models for code and test generation and constructs a binary pass/fail matrix $P \in \{0,1\}^{C \times T}$ at inference time, iteratively updating code and test scores by
$$
s_{\text{code}}^{(k+1)} = \frac{P \cdot s_{\text{test}}^{(k)}}{\mathbf{1}^T s_{\text{test}}^{(k)} + 10^{-8}}, \qquad
s_{\text{test}}^{(k+1)} = \frac{P^T \cdot s_{\text{code}}^{(k+1)}}{\mathbf{1}^T s_{\text{code}}^{(k+1)} + 10^{-8}}.
$$
LLMLOOP instead organizes refinement around a concrete Java/Maven project representation with source files, dependencies, tests, PMD reports, and PIT mutants, so that each loop can target a specific failure channel [2412.13464] [2603.23613].

## 3. Execution channels and feedback signals

Execution in GER is valuable only insofar as it yields actionable feedback. Several systems therefore emphasize observability rather than mere verdicts. The LLM-judge refinement tool exposes not only the judge’s categorical decision and rationale but also the exact generation and judging prompts through a “View explanation” affordance. The same interface stores generation prompts for each synthetic test case, including borderline cases synthesized from the current criterion definitions, so users can diagnose whether disagreement originates in the rubric, the prompt, or the case itself [2511.04478].

Other systems operationalize feedback through formal or runtime diagnostics. In loop-invariant synthesis, failures arise from proof obligations such as failed preservation, and the system records event penalties for abduction and conjecturing while using MCTS-guided trajectories to update solver and teacher policies. In PER-KBQA, execution returns intermediate entities or literals at each program step; refinement then conditions on those traces rather than on the raw logical form alone [2205.14229] [2305.03356].

Code-oriented GER work uses the full spectrum of software-engineering feedback. The iterative code-correction study employs compiler errors, runtime errors, failing test cases, expected-vs-actual outputs, and time or memory limit messages, and measures success with ISR@k and MIS; it reports that Wrong Answer and TLE account for about 95% of all failures, while Compile Error and Runtime Error are much more tractable [2606.17514]. OpenCodeInterpreter formalizes multi-turn execution feedback as execution outputs, exceptions, and optional synthetic human guidance inside dialogue state, and trains on 68K multi-turn Code-Feedback interactions [2402.14658]. LLMLOOP extends the feedback surface further by adding PMD static-analysis violations, JUnit failures, and PIT surviving mutants, each routed into its own refinement loop [2603.23613].

Agentic execution environments produce richer multimodal signals. ForenAgent receives heatmaps, spectra, scores, thumbnails, and runtime errors from a whitelisted Python toolbox, while AgentCyTE receives XSD validation errors, connectivity matrices, routing inconsistencies, segmentation violations, and service-state observations from CORE-based network emulation [2512.16300] [2510.25189]. This suggests that GER becomes more effective when execution returns structured diagnostics rather than a single scalar success flag.

## 4. Localization, reflection, and human oversight

A major line of development within GER is the move from whole-artifact rewriting to localized refinement. Reflect-SQL is explicit on this point: violations are localized to the responsible stage, and the update is applied to the stage parameterization $\theta_t$ rather than to the current SQL string. The stated goal is to maximize preservation of previously validated constraints and support monotonic improvement over a query set [2601.06678]. REFINE adopts the same logic for educational feedback: its judge produces binary component-level judgments and targeted explanations, and regeneration rewrites only the failing feedback components rather than the entire response [2603.29142].

Human-in-the-loop systems make localization concrete in the interface. In the LLM-as-a-judge tool, users inspect disagreements between expected and judged outcomes, revise criterion descriptions, generate targeted new cases, and use inline span-level operations—Paraphrase, Elaborate, Shorten, Regenerate—to perturb only the highlighted region while preserving surrounding context [2511.04478]. AlgoTouch does something analogous for imperative programs: a conditional may be created with an `else` branch left as `TODO`, and that branch is only completed when a later concrete execution reaches it; loops are assembled non-linearly by first recording a representative body, then termination conditions, then initialization [2606.03349].

Reflective behavior is especially prominent in agentic tool-use systems. ForenAgent’s loop includes global perception, local focusing, iterative probing, and holistic adjudication, with reflection triggered by incoherent cues or tool failure. The paper describes recovery actions such as recropping to an alternate ROI, switching tool families, or adjusting JPEG quality grids, all within a hard tool budget of $T_{\max}=7$ [2512.16300]. AgentCyTE similarly uses a hypothesis agent to generate multiple causal explanations of failure and multiple repair candidates, then selects the variant minimizing residual errors, thereby avoiding deterministic repair cycles [2510.25189].

These designs collectively shift GER from simple retry logic toward controlled revision. A plausible implication is that refinement quality depends less on the number of iterations than on whether the system can identify which subcomponent failed and preserve what is already correct.

## 5. Reported empirical outcomes

The empirical record is heterogeneous, but several papers report measurable gains from GER-style execution feedback or localized refinement.

| Setting | Reported outcome | Reference |
|---|---|---|
| Synthetic data for LLM judges | 83% preferred the tool; synthetic cases: $M=9.08$ vs $5.21$; final alignment $0.57$ vs $0.59$ and not significant | [2511.04478] |
| Loop invariant synthesis | Greedy solve rate on Code2Inv: random $18.4 \pm 0.0$, untrained teacher $39.7 \pm 1.6$, full teacher $61.5 \pm 0.4$; with MCTS the nondeterministic strategy solves 100% | [2205.14229] |
| GenX on APPS+ | Code pass@1 for Intro/Inter/Comp: $23.40/2.20/1.10$; ranked $1@10$: $30.60/4.00/2.00$; ranked $Pr@10$: $62.13/53.78/52.36$ | [2412.13464] |
| ForenAgent on FABench | Accuracy $88.1\%$, F1 $88.2\%$; removing Cold Start gives $79.8/78.7$ and removing RFT gives $81.9/81.7$ | [2512.16300] |
| Reflect-SQL | Spider: $93.8$ EX; BIRD: $65.2$ EX and $66.5$ VES; GPT-5 variant: Spider $95.4$ EX, BIRD $67.8$ EX and $68.9$ VES | [2601.06678] |
| LLMLOOP on HumanEval-X | pass@1 rises from $71.65\%$ to $80.85\%$; pass@10 rises from $76.22\%$ to $90.24\%$ | [2603.23613] |

The gains are not uniform in mechanism. The iterative code-correction study finds that reasoning models improve steadily across iterations, whereas non-reasoning models often plateau after one or two iterations; Compile Error is fixed about $88.2\%$ of the time and Runtime Error about $60.9\%$, but Wrong Answer is fixed only about $34.4\%$ and TLE about $20.0\%$ [2606.17514]. “Feedback Over Form” reaches a closely related conclusion at 1–3B scale: self-refinement with execution feedback improves code generation by more than 4 standard deviations on both HumanEval and MBPP, but the mechanism is narrow, fixing many NameError and SyntaxError cases while rarely fixing AssertionError, and every forced iteration is net-negative without early stopping [2604.21950].

The code-agent literature reports similar benefits from multi-turn execution traces. OpenCodeInterpreter-33B reaches an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, rising to 91.6 (84.6) with synthesized human feedback from GPT-4 [2402.14658]. In educational feedback, REFINE reports that judge-guided regeneration significantly improves feedback quality, and that the fine-tuned 8B interactive agent is not significantly different from GPT-5 on relevance, actionability, or tool relevance; its mean steps are 2.26 versus 2.39 for the 8B Thinking baseline [2603.29142].

Taken together, these results indicate that GER is most consistently beneficial when the execution layer provides precise, localizable signals and when refinement is constrained enough to preserve earlier correct structure.

## 6. Limitations, failure modes, and design tensions

The literature also records recurring limitations. Synthetic-data GER can plateau in diversity: the LLM-judge study notes repetition under the same configuration despite high temperature, “clean” outputs that lack real-world messiness, and practitioner concerns about circularity or model collapse when model-generated data is used for both generation and judging [2511.04478]. GenX reduces false positives by executing generated tests on ground-truth code and filtering code against accumulated tests, but its authors still identify diversity and representativeness as open questions, and pass@10 can drop relative to APPS even as pass@1 improves on APPS+ [2412.13464].

In search-based reasoning systems, refinement can be compute-intensive or theory-limited. Loop-invariant synthesis still relies on MCTS during training, and its current abduction engine is restricted to linear integer arithmetic; the authors explicitly note brittleness on out-of-distribution programs and the engineering effort required to design useful teacher constraints [2205.14229]. Reflect-SQL identifies non-convergence within budget, critic mislocalization, lock-in to incorrect beliefs, and incomplete semantic-checker coverage as failure modes, even though its stage-localized preservation strategy mitigates some drift [2601.06678].

Code GER highlights a separate tension between iteration and regression. The iterative code-correction study shows that wrong answers and TLE dominate the failure distribution and are harder to fix because they require algorithmic redesign rather than local patching [2606.17514]. “Feedback Over Form” makes the regression issue explicit: without early stopping, every iteration is net-negative, because refiners frequently damage already-correct code [2604.21950]. LLMLOOP adds additional validation channels, but at the cost of repeated LLM invocations and tool runs, and the paper explicitly notes that runs can take several minutes [2603.23613].

Execution-centered programming work exposes a different boundary. “A Systematic Approach to Programming” requires careful selection of traces until the control-flow graph and guards stabilize, and the underlying synthesis problem is formalized through restricted graph coloring rather than solved automatically in full generality [1808.08989]. AlgoTouch reduces search by making the user choose guards and delimit loop structure from concrete executions, but ambiguous guard selection, nondeterministic operations, and richer data structures remain out of scope [2606.03349].

Across domains, the same tension recurs: GER is strongest when execution feedback is informative, inexpensive, and localizable, but weaker when the feedback is sparse, the search space is too large, or refinement rewrites more than it preserves.

Source: https://www.emergentmind.com/topics/generate-execute-refine-loop