INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Abstract: Self-evolution offers a scalable path to stronger reasoning: a pretrained LLM improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

• The paper introduces optimizer-aware influence scores to create solver-specific curricula from unstructured documents.
• It demonstrates over 20% relative improvement in reasoning benchmarks across math, science, and coding by dynamically adapting the training data.
• The bilevel co-evolution of generator and solver enables adaptive curriculum learning that scales effectively across domains.

Influence-Guided Self-Evolution in INFUSER: An Expert Analysis

Introduction: Motivation and Core Problem

The INFUSER framework ("INFUSER: Influence-Guided Self-Evolution Improves Reasoning" (2606.09052)) addresses a fundamental bottleneck in LLM reasoning: the scarcity of high-quality, verifiable, and useful training data, particularly when progressing to advanced reasoning domains. While RL with verifiable rewards (RLVR) has yielded substantive improvements in LLM reasoning, it is ultimately constrained by the supply of suitable training data, which is expensive to curate or generate at scale. Prior self-evolution frameworks attempt to break this bottleneck by letting models generate their own training curriculum, often leveraging document pools or self-play. However, they suffer two critical deficiencies:

1. Grounding: Many rely on model outputs alone or limited formal anchors (e.g., code executors), leading to hallucination or domain restrictions.
2. Utility of Generated Data: The typical practice of rewarding generators according to task "difficulty" is a coarse heuristic and fails to guarantee that generated examples yield productive learning signals for the solver.

INFUSER fundamentally reconceptualizes this self-evolution loop by formulating it as a bilevel game between a generator (which drafts questions/answers from unstructured documents) and a solver (which trains on this curriculum). The key advance is the introduction of optimizer-aware influence scores, measuring the prospective utility of each generated example: namely, does training on this QA pair incrementally improve the solver on a target distribution? This mechanism enables an adaptive, solver-specific curriculum tightly aligned with desired learning progress.

Figure 1: INFUSER’s bilevel game views the generator as proposing curriculum from raw documents and the solver as training on it, with generator credit assigned by optimizer-aware influence scores based on dev-set alignment.

Methodology: Bilevel Curriculum Optimization and Influence Rewards

The formalization casts the generator and solver as co-evolving agents, both initialized from the same pretrained LLM but subsequently diverging through independent optimization. Each training iteration comprises the following sequence:

• Document Conditioned Generation: The generator, given sampled chunks from a diverse, open-source textbook pool, proposes multiple QA pairs per document.
• Solver RL Update: The solver trains via RLVR on these QA pairs, treating generator-provided answers as gold.
• Influence Reward Estimation: Using a small fixed development set sampled from the true target distribution, the system computes the gradient of solver performance with respect to solver parameters; each generated QA’s utility is then the cosine similarity (‘influence score’) between the per-sample update direction—preconditioned by the AdamW optimizer—and the dev-set improvement direction.
• Generator RL Update: The generator receives this scalar influence reward and is updated using a dual-normalized version of GRPO (DuGRPO), stabilizing the impact of high-variance and continuous rewards.

This structure ensures the curriculum is neither constrained by the model’s static prior (as in self-play) nor haphazardly aligned with general "difficulty," but instead shaped specifically to improve solver performance where it matters.

Figure 2: INFUSER’s detailed data flow: generator proposes self-generated QA from documents, solver answers and is rewarded, and influence scores feed generator RL updates. Dev-set gradients provide the optimization anchor.

Crucially, INFUSER’s generator is explicitly optimizer-aware: its reward is not the vanilla influence function, but a preconditioned variant that adapts to the momentum and scaling of AdamW, yielding more reliable credit assignment across variable-length examples and optimization regimes.

Empirical Results: Benchmarks, Ablations, and Curriculum Quality

Benchmark Performance and Scale

INFUSER exhibits robust, cross-domain improvements across a diverse suite of reasoning benchmarks—encompassing math (AIME, HMMT, OlympiadBench, MATH500), general science (SuperGPQA, GPQA-Diamond, MMLU-Pro, BBEH), medical, and coding domains. On the flagship Qwen3-8B-Base model, INFUSER delivers over 20% relative gain in aligned (math and science) benchmarks versus both the base model and strong recent baselines (R-Zero, AZR, SPICE, R-Few).

Figure 3: Relative accuracy gain over Qwen3-8B-Base on four headline benchmarks per self-evolution method, and validation-set accuracy curves showing INFUSER’s consistent, stable learning advantage over a frozen-generator baseline.

Notably, INFUSER’s gains persist and even scale as the base model increases in size (from 4B to 8B parameters), unlike other self-evolution methods where performance saturates or regresses at larger scales.

Generator Analysis and Curriculum Evolution

A distinctive feature of INFUSER is its ability to produce a curriculum of rising difficulty and quality tailored at the solver's current learning frontier. As training progresses:

• The accuracy gap (“strong-against-weak gap”) between a strong model (e.g., GPT-5.4) and the Qwen3-8B-Base on generated questions increases, evidencing the production of challenging, non-trivial examples without degenerate drift.
• Qualitative inspection shows marked improvement in question self-containment and factual alignment—the generator learns to pose nuanced, well-posed multi-step reasoning tasks not achievable by statically frozen generators.

Ablations: Role of Generator Adaptation and Reward Design

Comprehensive ablation studies underscore the necessity of several design decisions:

• Co-evolving Generator is Critical: A fixed or even large (“Strong-gen,” e.g., Qwen3-32B) frozen generator is less effective than an 8B generator co-evolving with the solver, especially for synthetic reasoning domains (math, coding).
• Dev-set Memorization vs. Generalization: Directly training the solver on the dev set achieves high accuracy only on the dev distribution (potentially overfitting), whereas INFUSER generalizes to held-out test sets, indicating that the explicit separation of influence evaluation and solver reward is essential.
• DuGRPO Advantage: The dual-normalized policy-gradient update for the generator mitigates the detrimental variance amplification present in standard GRPO/Dr.GRPO or batch-only alternatives, enabling stable generator learning atop noisy influence rewards.
• Optimizer-Awareness is Necessary: Substituting the AdamW-preconditioned influence score with a vanilla (SGD) gradient similarity led to regression to frozen-generator performance.

Figure 4: Head-to-head category Δ comparison of INFUSER vs. General-Reasoner and ablation settings, highlighting INFUSER’s superiority in mathematics and its balanced cross-domain transfer.

Hybridization with RLVR and Instruction-Finetuned Anchors

INFUSER retains flexibility: it augments instruction-finetuned models (e.g., OLMo-3-7B-Instruct-SFT) and hybridizes with math RLVR, stabilizing reasoning depth and sampling variance, particularly by anchoring reasoning length through verified math tasks.

Figure 5: pass@k curves on open-form math and general reasoning benchmarks, showing that INFUSER improves both best-of-k and support coverage, not only top-1 accuracy.

Theoretical and Practical Implications

INFUSER demonstrates the following:

• Data-Efficient and Targeted Self-Evolution: Influence-guided rewards enable LLMs to efficiently convert massive, cheap, unstructured corpora into highly targeted, impactful training signals—bypassing the need for large curated or teacher-generated datasets.
• Adaptive Curriculum Learning: By aligning the generator’s reward with actual incremental solver improvement (not heuristic difficulty), INFUSER learns to adapt the complexity of generated tasks as the solver advances, effectively “auto-curriculum learning” in a closed loop.
• Decoupling of Grounding and Reasoning: The separation of grounding (via document pools) and reasoning target (via dev set influence) affords broad flexibility and domain transfer, evident in the positive (albeit smaller) gains for medical and coding benchmarks, despite the primary focus on science/math.
• Solver-Generator Synergy Over Scale: The finding that a co-evolving generator can outperform far larger but frozen counterparts suggests fundamental limits in the efficacy of static data generation, and calls attention to the importance of solver-informed, adaptive data synthesis.

Theoretically, the adoption of first-order influence-guided curriculum optimization connects self-evolution for LLMs with contemporary meta-learning and dataset distillation paradigms, but at LLM scale and with domain-specific optimization for the RLVR setting.

Future Directions

While INFUSER's design points towards more effective unsupervised or weakly-supervised scaling, several open problems and avenues arise:

• Out-of-Domain Targeting: Extending effective influence-guided self-evolution to domains with minimal alignment between document pools and target dev sets.
• Question Quality and External Auditing: Incorporation of external tools or more sophisticated agentic generator loops to guarantee correctness and pedagogical value of synthesized questions.
• Dynamic Budget Allocation: Optimizing training budget allocation among multiple signal sources (e.g., science self-evolution vs. RLVR) for balanced cross-domain reasoning improvement.
• Scaling Document Pools and Training Horizons: Large-scale empirical studies exploring curriculum emergence with larger, more heterogeneous document corpora and longer co-evolution windows.

Conclusion

INFUSER represents a substantial methodological and empirical advance in self-evolution for LLM reasoning. By formulating curriculum generation as a bilevel game and introducing optimizer-aware influence rewards, INFUSER transforms raw unstructured documents into an adaptive, solver-specific curriculum that reliably and efficiently improves reasoning capability, outperforming state-of-the-art self-evolution and curated-dataset RLVR baselines. This approach promises to make high-level reasoning more accessible in LLMs without prohibitive data curation and to serve as a blueprint for future research in scalable, automated curriculum design for artificial intelligence.