Iterative Experience Refinement

Updated 26 April 2026

Iterative Experience Refinement (IER) is a cyclic process where autonomous systems continuously generate, evaluate, and refine candidate solutions based on external or internal feedback.
IER methodologies employ bandit-based algorithms, agentic inference techniques, and heuristic filters to balance exploration and exploitation in evolving solution spaces.
Empirical benchmarks reveal that IER strategies can achieve significant efficiency gains and improved success rates, reducing LLM calls and enhancing quality through selective experience retention.

Iterative Experience Refinement (IER) refers to a set of methodologies in which autonomous systems, especially LLM-driven software agents and agentic inference-time frameworks, continuously improve their task performance by repeatedly generating, evaluating, and updating candidate solutions or internal experiential knowledge through structured feedback and selective retention. Unlike static paradigms that utilize fixed caches of prior experience or one-shot outputs, iterative refinement mechanisms emphasize cyclic processes where new solutions (or experiences) are created, assessed according to external or internal criteria, and only high-value or frequently useful elements are retained for future use. This paradigm is foundational in tasks where problem solutions evolve through trial, error, and selective improvement, such as code synthesis, structured generation, and multi-turn planning (Qian et al., 2024, Tang et al., 2024, Chakraborty et al., 2 Apr 2025).

1. Core Principles and Formal Definitions

IER encompasses both explicit refinement of candidate solutions (e.g., code, plans, behaviors) and management of agentic experience pools (instruction–solution pairs or "shortcuts"). In formal terms, IER views the output space as dynamically expanding: each iteration yields new candidates or knowledge, with recursive evaluation and propagation.

In code synthesis applications, the process can be framed as an iterative search through candidate program spaces, where each step consists of generating a refinement and evaluating it with respect to an external specification $\Phi$ . Mathematically, the iterative process can be modeled as an arm-acquiring bandit problem, where each candidate program $p$ is an "arm." Let $A_t$ denote the active set of candidates at step $t$ , refined with respect to an external or learned reward signal $r_{t,p} \in \{0,1\}$ based on $\Phi$ (Tang et al., 2024).

For agentic software frameworks, IER reorganizes the agent's internal experience pool $\mathbb{E}$ by continuously harvesting new experiences from ongoing task executions and applying elimination heuristics, rather than relying on a fixed repository (Qian et al., 2024).

2. Representative Algorithms and Patterns

Several canonical patterns and algorithms instantiate IER:

REx Bandit-Based Refinement: REx formulates code repair as an arm-acquiring bandit problem, leveraging Thompson Sampling for arm selection. Each candidate's reward probability $\theta_p$ is modeled with a Beta posterior, updated using a heuristic $h(p)$ reflecting current success (e.g., test pass rate). Candidate selection thus balances exploitation (refining top-performing programs) against exploration (testing less-refined variants), guided by sampling from updated posteriors (Tang et al., 2024).
Iterative Agent Decoding (IAD): IAD structures inference-time agentic tasks into a cyclic process of candidate generation, verification with a scalar reward (possibly using an LLM as a judge), and explicit prompt re-conditioning on the best/worst candidates and feedback. IAD's pseudocode alternates between sampling, verifier-guided scoring, best/worst selection, and adaptive prompting, with convergence monitored by reward improvement (Chakraborty et al., 2 Apr 2025).
Successive and Cumulative Patterns for Experience Propagation: In multi-agent software development, IER can propagate experiences either successively (passing only immediate past batch experiences to the next batch) or cumulatively (amassing all prior experiences for future use). Formal definitions track $\mathbb{E}_i$ as the experience pool available at batch $p$ 0, with transitions governed by task executions and experience acquisition functions $p$ 1 (Qian et al., 2024).

3. Managing the Experience or Candidate Pool

The sustainability and efficiency of IER hinge on the selective retention and elimination of experiences or candidate solutions to avoid memory bloat and stale knowledge.

Heuristic Elimination Strategies are central to this:

Static Information-Gain Filter: Acquired shortcuts $p$ 2 are filtered by an information-gain metric $p$ 3, retaining those that surpass a threshold $p$ 4 in added value (Qian et al., 2024).
Dynamic Retrieval-Frequency Filter: Shortcuts are further filtered according to retrieval frequency, where only the top $p$ 5 fraction by usage frequency is retained (Qian et al., 2024).
Bayesian Prior Structuring and Posterior Updating: In bandit-based refinement, arm priors are modulated with task-specific heuristics and updated upon failures, ensuring rapid downgrading of low-quality arms and focus on high-potential candidates (Tang et al., 2024).

Resource Complexity is dependent on pool growth. The successive pattern maintains only one batch's experiences in memory, while the cumulative pattern's size grows with all batches, checked by elimination (e.g., pruning to ~11.5% of the full experience pool while maintaining performance) (Qian et al., 2024).

4. Exploration–Exploitation Tradeoffs

IER frameworks operationalize the inherent exploration–exploitation tradeoff in iterative problem solving:

Bandit Algorithms (REx): Refinement arms (candidate programs) face a tradeoff between further exploiting those likely to succeed (high heuristic score, few failures) and exploring lightly sampled candidates to avoid premature convergence to local optima. Thompson Sampling ensures diversity by assigning nonzero probability to all arms, but with bias towards promising ones (Tang et al., 2024).
Verifier-Guided Refinement (IAD): IAD's feedback mechanisms drive progressive improvement not achievable by one-shot sampling (e.g., Best-of-N). The reward-based selection loop continues to extract signal from scalar or LLM-judge feedback, with empirical convergence depending strongly on verifier accuracy and noise tolerance (Chakraborty et al., 2 Apr 2025).
Propagation Patterns: Successive propagation offers higher immediate gain but can be unstable, while cumulative patterns yield smoother improvement, highlighting a spectrum of tradeoffs as tasks or domains require (Qian et al., 2024).

5. Empirical Results and Benchmarks

IER approaches demonstrate quantifiable superiority over static or noniterative baselines across multiple domains:

Method/Domain	Notable Gains	Reference
REx (Bandit Refinement)	Up to 4–5× fewer LLM calls for each solved problem; 28/38 loop invariants solved vs. 24/38 by G-CLN; consistent AUC and efficiency improvement across ARC and APPS benchmarks	(Tang et al., 2024)
IAD (Agentic Inference)	3–6% absolute gains on Sketch2Code/Text2SQL tasks, 8–10% higher success rates on Webshop; improvement persists with sparse/noisy verifiers	(Chakraborty et al., 2 Apr 2025)
IER (Software Agents)	11% relative quality gain over static ECL baseline; experience elimination preserves quality with only 11.5% of original shortcut pool	(Qian et al., 2024)

Additional findings:

REx: Outperforms greedy, breadth-first, and fixed-width strategies, achieving higher area under the solved-vs-calls curve and robust performance with diverse prompt heuristics and test-case designs.
IAD: Surpasses Best-of-N at low sampling budgets and scales with both verifier fidelity and candidate pool size. Unlike BON, IAD enables monotonic improvement even when reward is sparse or noisy.
IER Patterns: Successive propagation reaches higher peak quality, while cumulative is more stable. Experience elimination is essential for scalability.

6. Practical Guidelines and Implementation Best Practices

Key recommendations extracted from empirical and algorithmic studies:

Heuristic Design: In REx, set $p$ 6 as fraction of passed test cases; weighted or domain-informed heuristics can further improve bias and convergence (Tang et al., 2024).
Hyperparameter Tuning: $p$ 7 balances exploitation and exploration in REx; $p$ 8 and $p$ 9 respectively filter out trivial shortcuts and long-tailed, low-utility experiences in IER (Tang et al., 2024, Qian et al., 2024).
Batching: Optimal batch size (e.g., 6 batches over 1,200 tasks) trades off experience diversity and refinement focus. Large batches encourage diversity; small batches more focused propagation (Qian et al., 2024).
Test Case Design and Prompting: Diverse test banks and randomized counterexamples accelerate refinement. For code tasks, concise prompts combining buggy code, failed case, and full specification are most effective (Tang et al., 2024).
Stopping Criteria: Either halt on first success or after a reasonable LLM call budget (e.g., 64 for ARC, 300 for APPS). Carefully calibrated budgets capture most marginal gains while controlling resource use (Tang et al., 2024).
Experience Elimination: Always apply after the pool exceeds several thousand entries to sustain hit rates and prevent system bloat (Qian et al., 2024).
Verifier Quality: In IAD, improvements scale with better verifiers; however, moderate noise and sparsity can be tolerated. Both IAD and BON break down with extreme noise but IAD is more robust to moderate imperfections (Chakraborty et al., 2 Apr 2025).

7. Theoretical and Methodological Connections

IER generalizes and extends across multiple research themes:

Reinforcement Learning Analogy: IER's generate–evaluate–refine cycle mirrors policy improvement in RL, with zeroth-order feedback replacing explicit gradient access; prompt re-conditioning effectively minimizes divergence $A_t$ 0 from an optimal (but inaccessible) policy (Chakraborty et al., 2 Apr 2025).
Bandit Theory: Arm-acquiring bandit models (Whittle, 1981) mathematically underpin the exploration–exploitation dynamics, providing Bayesian principled policies and analytical efficiency guarantees (Tang et al., 2024).
Experience Replay/Episodic Memory: Unlike standard buffer-based replay, IER incorporates dynamic experience acquisition and elimination customized to downstream utility, rather than uniform accumulation and retrieval (Qian et al., 2024).
Inference-Time Optimization: IER subsumes both black-box inference methods (e.g., Best-of-N sampling) and dynamic self-improvement based solely on runtime signal, without access to model parameters or inner-loop updates (Chakraborty et al., 2 Apr 2025).

A plausible implication is that as the field moves toward more autonomous, agent-driven problem solving (especially in software engineering and structured generation), frameworks grounded in IER will become central to scaling capabilities while maintaining efficiency and adaptability.