---
title: Reward-Based Self-Evolution
url: https://www.emergentmind.com/topics/reward-based-self-evolution
type: topic
---

# Reward-Based Self-Evolution

to=arxiv_search.search 彩票直属_json code 鲁夜夜啪ी{"query":"\"reward-based self-evolution\" OR \"self-evolution\" reward model language model multimodal arXiv", "max_results": 10}
to=arxiv_search.search  天天中彩票有人ి_json code  日本一本道{"query":"2604.03647 OR 2606.09052 OR 2604.18320 OR 2604.03098 OR 2411.00418", "max_results": 10}
to=arxiv_search.search  天天中彩票上ి_json code 
to=arxiv_search.search  天天中彩票官网י_json code  老时时彩{"query":"2604.03647", "max_results": 5}
Reward-based self-evolution denotes a family of closed-loop learning schemes in which a model or agent improves by generating, transforming, or selecting its own feedback signals and then reusing those signals for subsequent optimization. In the recent literature, this includes unsupervised pseudo-label reinforcement for multimodal reasoning, self-judging language-model training, self-evolving reward-model learning, co-evolving task generators and solvers, and policy–internal-reward co-training in long-horizon agents [2604.03647] [2606.09052] [2505.08827] [2604.03098]. The common objective is to reduce dependence on dense human annotation or static external reward models while preserving a sufficiently informative training signal to avoid collapse, reward hacking, or distribution drift.

## 1. Conceptual scope and historical framing

Across the cited work, reward-based self-evolution is not a single algorithmic template but a design space organized by where the reward comes from, what component evolves, and what remains externally grounded. Some systems evolve the policy by reusing its own judgments; some evolve the reward model itself; some co-evolve multiple roles such as Generator and Solver; and some attach reward-like signals to internal self-guidance or to latent structures such as prototypes [2401.10020] [2411.00418] [2606.09052] [2503.09394] [2604.03098].

| Mechanism locus | Representative mechanism | Example works |
|---|---|---|
| Policy self-judging | LLM-as-a-Judge, self-judging, Iterative DPO | [2401.10020], [2505.08827] |
| Reward model self-improving | synthetic preferences, self-labeling, on-policy reward recycling | [2401.12086], [2411.00418], [2509.22624] |
| Multi-role co-evolution | Generator/Solver, Challenger/Solver, Proposer/Solver/Judge | [2606.09052], [2604.18320], [2510.23595] |
| Internal reward co-evolution | self-generated step-level internal reward | [2604.03098] |
| Context-level reward evolution | Skills-and-Tools Library evolution | [2605.08703] |
| Latent-structure co-evolution | Prototype-Reward Interactive Evolution | [2503.09394] |

A broader historical antecedent appears outside contemporary LLM training. In the spatial public-goods game, adaptive rewarding was defined by a dynamic internal variable $\pi_x$ whose rewarding activity increases when defectors successfully invade and decays otherwise, making incentive intensity success-dependent and state-dependent [1208.3457]. This earlier formulation already instantiated a feedback loop in which the reward rule co-evolves with the underlying strategic population. By contrast, later work in language and multimodal modeling moves the same principle into post-training, curriculum generation, and learned evaluation.

The concept also has explicit boundary cases. Some papers argue that self-improvement need not rely on an explicit reward model at all: one line replaces reward-based optimization with environment-mediated selection, where only trajectories with $\Delta R(\tau)>0$ are retained for future supervised fine-tuning, while another replaces scalar external reward with empowerment and valence in a hierarchical planning architecture [2601.12310] [2211.10851]. These alternatives clarify that reward-based self-evolution is one branch of a wider inquiry into self-improving systems, not the only one.

## 2. Reward sources and mathematical forms

The modern literature uses several distinct reward sources. A first class uses **verifiable external signals**. EVE entirely bypasses pseudo-labels by executing Python image-transformation programs in a sandbox, deriving VQA answers from code execution, and rewarding the Challenger with format, validity, difficulty, and diversity terms [2604.18320]. SEEA-R1 combines MCTS-derived process rewards with a Multi-modal Generative Reward Model, while SEA assigns a Step Success Reward, Reasoning–Action Consistency Reward, and Action Format Reward at every action step in GUI trajectories [2506.21669] [2508.04037]. SPARK likewise starts from RLVR-style correctness checks and then recycles the same on-policy rollouts to train the model itself as a generative reward model [2509.22624].

A second class uses **self-judging**. In "Self-Rewarding Language Models" the model generates candidate responses, scores them with an additive 1-to-5 rubric under LLM-as-a-Judge prompting, and then uses those preference pairs for Iterative DPO [2401.10020]. In "Self Rewarding Self Improving" a fixed LLM judge returns binary correct/incorrect rewards without reference solutions, enabling GRPO on Countdown puzzles and MIT Integration Bee problems [2505.08827].

A third class uses **consensus-derived or self-labeled rewards**. CSRS replaces hard majority-vote pseudo-labeling with a Softened Frequency Reward. For answer $a$,
$$
f_{base}(a)=\frac{\mathrm{Count}(a,\mathcal{A}_{all})}{(m+1)n}, \qquad
f_r(a)=\frac{\mathrm{Count}(a,\mathcal{A}_m)}{m\,n},
$$
and
$$
R_{final}(a)=\bigl(\gamma\,\tanh\bigl(\beta\,(\tfrac{f_r}{f_{base}+\epsilon}-1)\bigr)+1\bigr)\times f_{base}.
$$
This replaces binary rewards with continuous signals in $(0,1)$ and is intended to damp over-concentration under repeated self-training [2604.03647].

A fourth class uses **influence-based or optimizer-aware rewards**. INFUSER rewards the Generator not for question hardness but for whether a generated question would actually improve the Solver on the target distribution. Its optimizer-aware influence score is
$$
s(q,a_\phi)=\mathrm{cosine}\bigl(\nabla_\theta J(\theta),\Gamma(q,a_\phi)\bigr),
$$
and Generator updates use DuGRPO, whose dual-normalized advantage is
$$
A_{gen}(q_i)=\frac{s(q_i)-\mu_d}{\sigma_d+\sigma_{\mathcal B}+\epsilon}.
$$
The stated purpose is to stabilize training under continuous, noisy influence scores [2606.09052].

A fifth class uses **internal rewards derived from self-guidance**. Self-Guide has the agent generate a natural-language self-assessment $z_t$ before each action, map it to $r_t^{sg}\in\{+0.1,0,-0.1\}$, and combine it with sparse environment reward through
$$
R(\tau;u)=R_{\mathrm{env}}(\tau)+\lambda(u)\sum_{t=1}^T r_t^{sg}.
$$
The same signal serves at inference time as guidance and at training time as dense internal reward [2604.03098].

Finally, RewardHarness departs from weight optimization altogether. It treats reward modeling as **context evolution**: an Orchestrator grows, prunes, and refines a library of Skills and Tools, and a frozen Sub-Agent uses the selected context to score candidate edits and produce preference judgments [2605.08703]. This suggests a variant of reward-based self-evolution in which what evolves is the evaluation context rather than the model parameters.

## 3. Structural archetypes of self-evolving systems

One recurrent architecture is **single-model self-rewarding**. In self-rewarding DPO, the same LLM acts as actor and judge across iterations $M_0,M_1,\dots,M_T$, so instruction-following ability and reward-modeling skill improve together [2401.10020]. The self-judging GRPO setting is similar, except that the judge may remain fixed while the policy learns from its binary outputs [2505.08827].

A second archetype is **role-separation with coupled optimization**. INFUSER maintains separate Generator and Solver policies initialized from the same pretrained checkpoint but with distinct parameters and optimizer states [2606.09052]. EVE alternates Challenger training and Solver training, using a priority queue of high-reward transformation programs to diversify the Challenger’s few-shot context [2604.18320]. MAE instantiates Proposer, Solver, and Judge from a single base LLM and applies synchronized Task-Relative REINFORCE++ updates across all three roles [2510.23595].

A third archetype is **policy–reward co-evolution** in embodied or interactive settings. ROSKA evolves reward functions via an LLM and policies via hybrid initialization plus SC-BO, selecting the most capable and plastic reward-policy combination each round [2412.13492]. Self-Guide unifies policy and internal reward generator inside one policy $\pi_\theta$, with the same model producing both $z_t$ and $a_t$ [2604.03098].

A fourth archetype is **reward–representation co-evolution**. BPRE maintains learnable visual prototypes $\{v_c\}$ and a Multi-Dimensional Quality-Aware Reward Module. Reward scores update prototypes via a weighted momentum rule, and updated prototypes feed back into the similarity reward for the next batch [2503.09394]. The object of evolution here is not a generative policy alone, but the coupling between feature evaluation and latent class anchors.

A fifth archetype is **agentic reward construction**. RewardHarness uses an Orchestrator for routing and evolution, a frozen Sub-Agent for reasoning, and a versioned external library of Skills and Tools [2605.08703]. The resulting system is self-evolving in the sense that reward competence improves through artifact revision rather than gradient updates to the evaluator backbone.

## 4. Stability problems and the mechanisms designed to prevent them

The dominant failure mode in pseudo-label-based self-evolution is self-reinforcing bias. CSRS identifies confirmation bias and distributional shift, termed “Model Collapse,” together with sampling sparsity in long-chain reasoning; repeated majority voting drives the answer distribution toward a narrow mode and suppresses rare but correct reasoning paths [2604.03647]. Its response is threefold: the Retracing Re-inference Mechanism revisits anchor points at fraction $\omega$ of a trajectory, Softened Frequency Reward replaces binary rewards with continuous signals, and Visual Semantic Perturbation adds Gaussian noise to the image during re-inference to force reliance on mathematical logic rather than visual superficiality. The paper reports that removing SFR or RRM causes drops of up to 2–3%, VSP removal costs about 1%, $m=5$ is optimal, and $\omega=0.7$ is robust [2604.03647].

In influence-guided co-training, the failure mode is noise amplification under continuous rewards. INFUSER states that standard GRPO can amplify noise when group standard deviation is small, which motivates DuGRPO’s mixture of within-group and batch-level normalization [2606.09052]. The paper’s ablations attribute performance to co-evolving the Generator, dev-set influence rather than pure dev-set memorization, optimizer-aware influence rather than raw gradients, and dual normalization rather than group-only or batch-only schemes.

In self-judging systems, the central risk is reward hacking and calibration drift. "Self Rewarding Self Improving" reports that weaker judge prompts on Countdown are quickly exploited, whereas only the strongest prompt maintained high true-negative rates under adversarial policy updates [2505.08827]. The judge is therefore kept fixed and offline, and an external formal verifier is used only for monitoring TNR and true reward curves. "Self-Rewarding Language Models" notes the risk that erroneous model evaluations may be amplified, that synthetic prompts may drift from genuine user needs, and that a self-judge cannot independently detect its own blind spots without external auditing [2401.10020].

Long-horizon agent training introduces a different stabilization problem: immature internal reward can bias the policy before it becomes trustworthy. Self-Guide addresses this with a trapezoidal Stage-Wise Trust Schedule: Phase I uses guidance only with $\lambda=0$, Phase II ramps $\lambda$ from 0 to 1, Phase III uses full internal reward, and Phase IV anneals back to $\lambda=0$ so that the final policy aligns with $R_{\mathrm{env}}$ alone [2604.03098].

Embodied self-evolution adds the need for denser intermediate signals. Tree-GRPO uses MCTS node-level process rewards as group-normalized advantages, while SEA uses step-wise rewards at every action step instead of a single terminal reward [2506.21669] [2508.04037]. EVE addresses stability from a different angle by moving supervision outside the model’s own certainty: correctness is fixed by sandboxed execution rather than by pseudo-label confidence [2604.18320].

## 5. Empirical domains and representative quantitative outcomes

The empirical record spans multimodal reasoning, instruction following, reward modeling, image-edit evaluation, embodied interaction, robotic control, and test-time adaptation.

| System | Domain | Reported outcome |
|---|---|---|
| CSRS | MathVision, MathVerse, MathVista, We-Math | gains of +2.34%, +1.48%, +2.41%, +3.28% over MM-UPT [2604.03647] |
| INFUSER | MATH500, OlympiadBench-Math, MMLU-Pro, SuperGPQA | 17–20% relative accuracy over strong self-evolution baselines [2606.09052] |
| Self Rewarding Self Improving | MIT Integration Bee | Qwen 2.5 7B climbs from initial ~35% to 43%, an 8% absolute gain [2505.08827] |
| RewardHarness | EditReward-Bench, GenAI-Bench | 47.4% average ranking accuracy using 0.05% of EditReward data [2605.08703] |
| Self-Rewarding Language Models | AlpacaEval 2.0 | win rate vs GPT-4 Turbo 9.94% → 15.38% → 20.44% [2401.10020] |
| EVE | ALFWorld-style multimodal self-evolution is not the setting; VQA benchmarks | Qwen3-VL-8B overall accuracy 66.48 → 68.7 over 5 iterations [2604.18320] |
| Self-Guide | ALFWorld | GRPO baseline 86.7% → 96.9% with SG$^{full}$ [2604.03098] |
| SEEA-R1 | ALFWorld | 85.07% textual, 36.19% multi-modal, and 80.3% without environmental reward [2506.21669] |
| ROSKA | Isaac Gym tasks | average normalized improvement of 95.3% across tasks [2412.13492] |

Several patterns recur in these results. First, **continuous or structured rewards** often outperform hard binary selection. CSRS replaces majority-vote 0/1 rewards with a softened frequency signal and reports higher policy entropy and broader semantic exploration in t-SNE plots [2604.03647]. SELF-EMO uses a smoothed multi-label IoU reward plus a group-level consistency reward and reports average accuracy improvements of +6.33 on Qwen3-4B and +8.54 on Qwen3-8B over raw backbones on IEMOCAP, MELD, and EmoryNLP [2604.18003].

Second, **self-evolving reward models** can reduce dependence on human annotation. SER begins with only 15% of the human-labeled RM portion and reports an average 7.88-point RM-accuracy improvement over the Loop 0 seed while approaching the Full RM with an average gap below 0.3 points [2411.00418]. West-of-N likewise reports +1–2% absolute accuracy on held-out human preferences and +4–5% boost in Best-of-64 generation win rate over the supervised checkpoint by augmenting reward-model training with synthetic Best/Worst-of-$N$ preference pairs [2401.12086].

Third, **agentic or context-level reward evolution** can be data-efficient. RewardHarness uses 100 preference demonstrations, evolves approximately 70–80 cycles, grows the library to about 13 entries and then prunes to 3 Skills plus 4 Tools, reaching 62.5% held-out ranking-agreement accuracy versus a 42.5% empty-library baseline [2605.08703].

## 6. Boundaries, controversies, and open directions

The literature does not treat reward as unproblematic. Several papers explicitly argue that externally fixed or easily manipulable proxies are a central source of collapse. "Survival is the Only Reward" replaces reward maximization with environment-mediated selection under real resource constraints and presents this as making proxy optimization impossible and reward-hacking evolutionarily unstable [2601.12310]. "Reward is not Necessary" rejects reward-maximization in favor of empowerment and valence as the organizing principle of life-long self-preserving behavior [2211.10851]. These works define the outer boundary of the field: reward-based self-evolution is powerful, but it is not universally regarded as necessary.

Even inside the reward-based camp, the main controversy concerns **where trustworthy feedback comes from**. Verifiable code execution in EVE, formal verification in Countdown and symbolic math, and environment rewards in ALFWorld provide strong anchors [2604.18320] [2505.08827] [2506.21669]. Self-judging, consensus-based pseudo-labels, and self-evolved reward models offer scalability but inherit risks of confirmation bias, judge drift, and reward hacking [2604.03647] [2401.10020] [2411.00418]. RewardHarness and Self-Guide occupy intermediate positions: one externalizes evaluation knowledge into editable artifacts, and the other densifies sparse reward with self-generated guidance while annealing back to environment reward [2605.08703] [2604.03098].

A second controversy concerns **what is actually evolving**. In some systems the policy evolves; in others the generator, reward model, prototype set, task distribution, or evaluation context evolves. This suggests that “reward-based self-evolution” is best understood as a family resemblance term for closed loops in which reward design and model improvement are no longer separable. A plausible implication is that future systems will increasingly combine verifiable anchors, self-generated curricula, and co-evolving evaluators rather than relying on any single reward source. That possibility is already visible in hybrids such as INFUSER augmented with rule-verifiable RLVR data, SEEA-R1 operating without environmental reward, and SPARK recycling RLVR rollouts to train an internal generative reward model [2606.09052] [2506.21669] [2509.22624].

Open problems stated across the literature remain substantial: judge reliability and plateau effects in fixed self-judging loops, extension beyond 2D PIL transformations in EVE, adaptive data-source allocation in hybrid curricula, uncertainty calibration for learned reward models, and transfer to domains lacking deterministic or programmatic checkers [2505.08827] [2604.18320] [2606.09052] [2509.22624]. The field’s central technical tension is therefore stable: maximize autonomy in reward generation without sacrificing the external grounding needed to keep self-evolution informative rather than self-confirming.

Source: https://www.emergentmind.com/topics/reward-based-self-evolution