Self-Evolving Pairwise Reasoning (EvolvR)

• Self-Evolving Pairwise Reasoning (EvolvR) is a framework that improves model reasoning by generating, critiquing, and reusing its own reasoning traces through pairwise comparisons.
• The method employs multi-persona synthesis and multi-agent filtering to refine chain-of-thought data, ensuring robust and logically coherent evaluations.
• EvolvR integrates closed-loop self-evolution with reward modeling and memory retrieval across domains, enabling continuous improvement and adaptation in complex tasks.

Self-Evolving Pairwise Reasoning (EvolvR) denotes a family of reasoning and training schemes in which a model improves by generating, comparing, critiquing, and reusing its own reasoning traces, policies, or tasks through pairwise or contrastive signals. In the framework explicitly named EvolvR for story evaluation, the method is grounded in pairwise comparison, first self-synthesizes score-aligned Chain-of-Thought (CoT) data via a multi-persona strategy, then applies a self-filtering process utilizing multi-agents to guarantee logical rigor and robustness, and finally deploys the trained evaluator as a reward model to guide story generation (Wang et al., 8 Aug 2025). Closely related systems instantiate the same motif through challenger–solver co-evolution, solver–verifier interaction, pairwise Elo evaluation, contrast between successful and failed trajectories, or memory-aware test-time scaling, yielding closed-loop systems that update reasoning behavior over time (Huang et al., 7 Aug 2025, Yu et al., 9 May 2026, Ouyang et al., 29 Sep 2025).

1. Conceptual basis and pairwise formulation

EvolvR is defined by two coupled commitments: reasoning is treated as an object that can be inspected and revised, and improvement is driven by pairwise or contrastive judgments rather than solely by pointwise labels. In the story-evaluation setting, the evaluator is defined over a pair of stories and an evaluation aspect,

$R_\phi(S_a, S_b, k) = (C_k; y_{a, k}; y_{b, k}),$

where the model produces both a rationale and pairwise scores. The underlying motivation is explicit: human agreement is higher on pairwise preferences than on pointwise absolute scores, and pairwise correlation is +21.9% higher for coherence and +12.5% higher for relevance on the HANNA dataset (Wang et al., 8 Aug 2025).

The same pairwise logic appears in algorithmic discovery. In Evolutionary Ensemble (EvE), agent quality is measured not in isolation, but by the marginal improvement their new solver brings over an identical baseline. A synchronous round produces a win–loss matrix over agents, and Elo rating updates impose selection pressure on strategies that are useful in the current search landscape rather than globally fixed ones (Yu et al., 9 May 2026). In this formulation, pairwise reasoning is not merely an evaluation convenience; it is the mechanism by which a population of guidance states adapts to changing optimization stages.

R-Zero generalizes the pairwise scheme into a co-evolutionary game between a Challenger and a Solver. The Challenger is rewarded for proposing tasks near the edge of the Solver capability, ideally where the Solver has a 50% chance of solving, and the Solver is then trained on the filtered self-generated curriculum (Huang et al., 7 Aug 2025). Here the pairwise relation is not story versus story or agent versus agent, but task proposer versus task solver, with uncertainty itself serving as the training signal.

Framework	Pairwise object	Update signal
EvolvR	Story pair $(S_a, S_b)$	Score-aligned CoT and reward modeling
EvE	Agent $a_i$ versus agent $a_j$	Win–loss matrix and Elo updates
R-Zero	Challenger versus Solver	Frontier difficulty and uncertainty reward

A plausible implication is that “pairwise” in EvolvR is best understood as a structural property rather than a single architecture: the system learns by contrasting alternatives under a shared context, and the contrast is preserved long enough to alter future behavior.

2. Core mechanisms: synthesis, filtering, memory, and retrieval

The original EvolvR pipeline for story evaluation is organized into three stages: Self-Synthesis of Score-Aligned Chain-of-Thought (CoT) Data, CoT Evolution and Multi-Agent Selection, and Deployment as a Reward Model. The synthesis stage uses multiple personas—academic, artistic, sharp-tongued, casual, pragmatic—to produce diverse, score-aligned rationales. The filtering stage then applies a sequence of agents: Self-Rule Agent, Self-Refinement Agent, Self-Rule Agent (again), Self-Counter (Self-Attack) Agent, and Self-Confidence Agent. The attrition is substantial: Of 800,000 candidates, only 67% pass all filters (to 536,177) (Wang et al., 8 Aug 2025). This design makes the self-generated rationale itself the object of quality control.

ReasoningBank moves the same logic from pairwise judgment to persistent reasoning memory. Each memory entry comprises Title, Description, and Content, and the source of memory includes both successful and failed task trajectories. After each task, the agent Labels its trajectory as success/failure, Distills reasoning strategies from the trajectory, and Stores the resulting memory items into a structured memory pool. At the start of a new task, it retrieves the most relevant memory items through embedding-based semantic similarity search over memory titles, descriptions, and content, with an example implementation using gemini-embedding-001 and cosine similarity (Ouyang et al., 29 Sep 2025). The critical shift is from replaying raw traces to reusing distilled principles.

EvolveR formalizes the same closed loop as an experience-driven lifecycle with Offline Self-Distillation and Online Interaction. Its internal experience base $\mathcal{E}$ stores principles as a tuple of natural language description and JSON-structured (subject, predicate, object) triples. Deduplication combines embedding similarity with LLM-powered semantic equivalence judgment, while online interaction introduces explicit action types—search_experience, search_knowledge, and answer—so that the agent can retrieve and apply prior principles during multi-hop reasoning (Wu et al., 17 Oct 2025).

Across these systems, factual commonalities are stable. Self-generated reasoning is not accepted verbatim; it is distilled, filtered, scored, or clustered. The reusable unit is not necessarily a final answer but may instead be a rationale, a principle, a heuristic, a memory item, or a guidance state.

3. Closed-loop self-evolution across agents, tools, and domains

ReasoningBank extends the memory loop into Memory-aware Test-Time Scaling (MaTTS). Parallel MaTTS generates $k$ independent rollouts for a query, while Sequential MaTTS iteratively refines a single solution and accumulates memory from all intermediate steps. In both cases, the expanded trajectories are used as a rich source of contrastive experiences, and the paper describes a powerful synergy between memory and test-time scaling in which better memory guides more effective scaling and diversified scaling yields stronger memory items (Ouyang et al., 29 Sep 2025).

EvE implements self-evolution as dual co-evolving populations: a solver population $\mathcal{S}$ and an agent population $\mathcal{A}$. In its ICON search setup, 2 working agents run in parallel, supported by 8 reference solvers and 4 reference agents, and pairwise Elo rating updates use $K=32$ (Yu et al., 9 May 2026). The framework’s central claim is that stage-dependent agent adaptation is indispensable: a static initial agent plateaus early, while a frozen best-evolved agent can underperform because of phase mismatch.

RareAgent translates self-evolving pairwise reasoning into biomedical discovery. Its four agents—Principal Investigator (PI), Explorer, Proponent, and Skeptic—conduct task-specific adversarial debates over a shared Task-specific Evidence Graph (T-EGraph) whose edges are labeled supports, refutes, or entails. After each run, the PI generates a Credit Assignment Report, revises prompts, and distills recurring heuristics into a shared policy library for future tasks (Qin et al., 7 Oct 2025). The pairwise component is concentrated in the Proponent–Skeptic adversarial debate, while the self-evolving component lies in policy revision and heuristic transfer.

Agent0-VL moves the same pattern into tool-integrated multimodal reasoning. A single LVLM unifies a Solver and a Verifier, and the two roles interact through a Self-Evolving Reasoning Cycle in which tool-based verification, confidence-gated self-repair, and GRPO jointly align reasoning and evaluation distributions. The framework reports an 12.5% improvement over the base model on geometric problem solving and visual scientific analysis (Liu et al., 25 Nov 2025). The distinctive feature is that tool use is incorporated not only into reasoning but also into self-evaluation and self-repair, addressing failures of purely text-based self-evaluation on visual tasks.

MathSE describes an adjacent closed-loop design for multimodal mathematics. It alternates inference, ORM evaluation, reflection/correction, and fine-tuning, where the Outcome Reward Model (ORM) identifies the precise error step and provides detailed error analysis for incorrect paths (Chen et al., 10 Nov 2025). This places error localization, rather than final-answer judgment alone, at the center of self-evolution.

4. Empirical record

Reported gains span open-ended evaluation, web agents, mathematical reasoning, multimodal reasoning, algorithm discovery, and biomedical discovery. The figures below are those explicitly reported in the cited papers.

System	Domain	Reported outcome
EvolvR (Wang et al., 8 Aug 2025)	Story evaluation and generation	SOTA on StoryER, HANNA, and OpenMEVA; win-rate against base model: up to 64.4%
ReasoningBank (Ouyang et al., 29 Sep 2025)	Web browsing and software engineering	up to 34.2% relative gain vs. no memory; up to 16% fewer actions
EvE (Yu et al., 9 May 2026)	ICON algorithmic discovery	Best $\overline{e}$ at 2k steps: 0.114, 0.108 and at 10k steps: 0.041, 0.045 for EvE (run 1,2)
RareAgent (Qin et al., 7 Oct 2025)	Rare-disease drug repurposing	Improves the indication AUPRC by 18.1% over reasoning baselines
R-Zero (Huang et al., 7 Aug 2025)	Zero-data reasoning	Boosts Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks
Prism (Mishra, 3 Mar 2026)	Mathematical self-evolution	+3.98 absolute points over R-Zero on AMC and +3.68 on Minerva Math
EvoEnv (Shi et al., 14 May 2026)	Zero-data reasoning RL	On Qwen3-4B-Thinking, improves 72.4 to 74.8, a relative gain of 3.3%
C2-Evo (Chen et al., 22 Jul 2025)	Multimodal mathematical reasoning	Geo-Sub-Aux: 40.4% → 54.9%; MathVista(ALL): 60.0 → 63.4
DSER (Liu et al., 20 Oct 2025)	AIME 2024–2025	Solves 5 out of 9 previously unsolvable problems; 82.8% → 89.3% on AIME 2024 and 74.4% → 83.4% on AIME 2025
MathSE (Chen et al., 10 Nov 2025)	Multimodal mathematical reasoning	Full GPT-4o: 58% on MathVL-test; Self-evolving MathSE: 62.35%

The empirical pattern is heterogeneous but coherent. Pairwise and self-evolving methods appear in two broad modes. One mode improves evaluators or reward models in open-ended settings, as in story evaluation. The other mode improves solvers, curricula, or training environments in domains with stronger verification, such as mathematics, coding, or executable environments.

A plausible implication is that EvolvR-style systems are most empirically stable when the pairwise signal can be grounded in durable structure: executable scorers, deterministic tasks, tool outputs, or repeated comparisons under matched conditions.

5. Limits, failure modes, and contested assumptions

The strongest negative result is the documented generalization gap in strict closed-loop self-evolution. Under a setup where the algorithm has access only to an unlabeled prompt set and a base model, self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. On Knights and Knaves, Gemma-3-4B-it improves from 31.0% to 40.7% under SimpleSE, $(S_a, S_b)$0, while an Oracle Verifier reaches 46.6%; after SimpleSE × 3, performance is 44.1%, and a Final Oracle round (after SE) reaches 53.2% (Qi et al., 31 May 2026). The same study reports that Pass@1 improves, Pass@32 does not, indicating sharpening of confidence rather than broadening of solution coverage.

A second limitation is curriculum collapse. Prism argues that prior self-evolving systems can exhibit diversity collapse after only a few iterations, even when surface-level variation is preserved. Its quantitative contrast is explicit: R-Zero covers 65/128 clusters, whereas Prism covers 107/128 clusters; the corresponding entropy is 3.71 versus 5.81, and the Gini is 0.90 versus 0.66 (Mishra, 3 Mar 2026). The claim is not that self-generated curricula necessarily diversify on their own, but that persistent semantic coverage must be enforced.

A third limitation concerns verifier quality and refinement stability. DSER argues that the effectiveness of many verification-refinement pipelines hinges on strong verification and correction capabilities that remain fragile in open-weight, smaller-scale models. It models iterative reasoning as a Markov chain with improvement probability $(S_a, S_b)$1 and degradation probability $(S_a, S_b)$2, and states that convergence to a correct solution is guaranteed as long as the probability of improvement marginally exceeds that of degradation (Liu et al., 20 Oct 2025). The same paper reports that a verification-dependent approach solved only 2 of the 9 hard AIME problems on the tested 8B model, whereas DSER solved 5 of these 9. This does not eliminate the role of verification; it narrows the requirement from perfect stepwise correction to a positive long-run improvement tendency.

Additional failure modes recur across the literature. EvE identifies phase mismatch when a frozen “best-evolved” agent is applied outside the stage for which it evolved (Yu et al., 9 May 2026). R-Zero documents a pseudo-label quality tradeoff, with correctness per GPT-4o dropping from 79% to 63% as the Challenger generates increasingly difficult questions (Huang et al., 7 Aug 2025). Taken together, these results caution against reading “self-evolving” as synonymous with unbounded autonomous improvement.

6. Relation to adjacent paradigms

EvolvR is closely related to pairwise reward modeling and search-based reasoning. LLaMA-Berry combines SR-MCTS, a Pairwise Preference Reward Model (PPRM), and Enhanced Borda Count (EBC) to compare and rank entire reasoning paths in Olympiad-level mathematics (Zhang et al., 2024). Although it is not framed as a lifelong agent system, it shares the same premise that reasoning quality is more robustly learned or selected through pairwise comparisons of complete trajectories than through local or greedy scoring.

A second adjacent direction replaces pairwise task comparison with environment construction. EvoEnv argues that stable self-improvement depends on solve--verify asymmetry, and instantiates a single-policy generator, solver method that synthesizes Python environments from ten seeds and admits them only after staged validation, semantic self-review, solver-relative difficulty calibration, and novelty checks (Shi et al., 14 May 2026). This relocates self-evolution from “generate more traces” to “construct reusable executable environments.”

A third direction generalizes co-evolution beyond text reasoning. C2-Evo jointly evolves multimodal training data and model capability through a cross-modal data evolution loop and a data-model evolution loop, while MathSE introduces self-evolving iterative reflection with an ORM for stepwise error analysis (Chen et al., 22 Jul 2025, Chen et al., 10 Nov 2025). These systems preserve the central EvolvR intuition—reasoning improves when the model repeatedly confronts and revises its own boundary cases—but they do so through data complexity alignment and reward-guided reflection rather than solely through pairwise judging.

In this broader landscape, EvolvR names both a specific framework and a broader methodological family. Its defining idea is that reasoning improvement can be bootstrapped from internal comparisons—story against story, trajectory against trajectory, success against failure, agent against agent, or solver against challenger—provided that the comparison is reliable enough to alter future behavior. The current literature shows that this principle is productive, but also that its success depends on verifier recall, curriculum diversity, stage-dependent adaptation, and the durability of the supervision signal (Wang et al., 8 Aug 2025, Qi et al., 31 May 2026, Mishra, 3 Mar 2026).