Guided Self-Evolving LLMs with Minimal Human Supervision (2512.02472v1)

Abstract: AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.

• The paper introduces the R-Few framework that integrates minimal human supervision using in-context anchors and curriculum-driven self-play to stabilize and enhance LLM evolution.
• It employs a Challenger–Solver architecture that curbs semantic drift and reward hacking while ensuring robust improvements with only 1%-5% of human-labeled data.
• Experimental results show that R-Few achieves significant gains in math and general reasoning tasks, outperforming baseline models by delivering +3.0 average performance improvements.

Guided Self-Evolution of LLMs with Minimal Human Supervision

Introduction

The paper "Guided Self-Evolving LLMs with Minimal Human Supervision" (2512.02472) addresses the persistent challenges in self-evolving LLMs, particularly the instability and early performance plateauing seen in unguided self-play paradigms such as R-Zero and Absolute Zero. Motivated by the need for stable, scalable reasoning capabilities while minimizing dependence on large-scale human annotations, the authors introduce R-Few: a meta-framework that strategically integrates lightweight human anchoring with curriculum-driven self-play. The central thesis is that judicious, minimal human supervision—implemented via in-context "anchor" examples and dynamic curriculum selection—can yield robust, data-efficient self-evolution for LLMs, mitigating issues like semantic drift, diversity collapse, and reward hacking.

Methodology

R-Few comprises two tightly coupled modules: the few-shot grounded Challenger and the online curriculum Solver. Both agents are initialized from a shared pretrained base LLM (e.g., Qwen3-8B-Base).

The Challenger generates synthetic reasoning problems conditioned on $k \in \{0, \ldots, 5\}$ contextually sampled human-annotated anchor examples. This few-shot anchoring regulates the semantic locality of generated tasks, deterring concept drift and encouraging exploration adjacent to human knowledge. The Challenger’s reward is shaped by a combination of uncertainty (targeting “medium-difficulty” samples at the capability frontier of the Solver) and similarity to the human anchor pool, augmented with penalties for duplicative or invalid questions.

The Solver performs multi-rollout evaluations to estimate its own success distribution over generated and human problems. An online curriculum filter selects samples in a designated uncertainty quantile (empirically, $[0.3, 0.7]$), establishing a self-adaptive “zone of proximal learning.” The Solver’s reward upweights human anchor data, preventing catastrophic forgetting of ground-truth semantics.

Iterations of Challenger and Solver updates constitute an iterative co-evolutionary training loop, with both RL objectives unified under Group Relative Policy Optimization (GRPO). The result is a continuous evolution of solving and proposing abilities, but with regulation and stability introduced by anchor grounding and curriculum curation.

Figure 1: Overview of the R-Few architecture. The Challenger is incentivized to produce medium-uncertain, frontier questions guided by human anchors, and the Solver improves via curriculum-guided training on both synthetic and human-labeled data.

Experimental Results

Performance Gains and Data Efficiency

R-Few was benchmarked on math (AMC, MATH-500, GSM8K, Minerva, Olympiad-Bench) and general reasoning (MMLU-Pro, SuperGPQA, GPQA-Diamond, BBEH) tasks. With only 1%–5% of human-labeled WebInstruct data as anchor, R-Few achieves performance on par with General-Reasoner, a model trained with approximately 20× more human annotations, and surpasses both R-Zero and Absolute Zero baselines. Quantitatively, Qwen3-8B-Base trained with R-Few (5%) delivers average gains of +3.0 over R-Zero in math domains; the 8B parameter scale consistently benefits more from guided self-evolution than smaller baselines. Notably, R-Few (5%) on Qwen3-8B-Base outperforms General-Reasoner on several metrics despite utilizing drastically less supervised data.

Figure 2: R-Few delays the performance plateau endemic to R-Zero and achieves higher, more stable self-evolutionary improvements across benchmarks.

Stability and Reward Hacking Mitigation

Self-play LLM training is susceptible to metric exploitation—models may artificially boost task diversity via verbosity or generate ungrounded, longer samples that score highly on difficulty. The authors measure diversity (2-gram lexical diversity), mean question length, and empirical difficulty (independent accuracy on generated tasks by ground-truth models).

R-Few stability is substantiated by:

• Sustained question diversity, as opposed to R-Zero’s collapse.
• Consistent question length, absent of superficial inflation.
• Difficulty increases attributable to genuine content rather than reward hacking.

Figure 3: R-Few maintains stable diversity and regular question length throughout training, whereas R-Zero exhibits diversity collapse and length inflation (reward hacking).

Ablations and Domain Analysis

Ablation studies confirm that removing Challenger training, Challenger warm-up (via SFT), or curriculum learning each leads to meaningful degradation, especially in mathematically intensive domains—emphasizing the necessity of each component. Further, targeted domain sampling of anchor data reveals strong on-domain improvements and beneficial transfer, especially between mathematically related fields (e.g., math-physics), permitting controllable growth trajectories through domain-specific anchoring.

Theoretical and Practical Implications

R-Few offers several substantive advances:

• Efficient data utilization: Minimal human supervision suffices for on-par or superior performance to heavily annotated regimes.
• Mitigated collapse: Lightweight anchoring and curriculum filter out spurious or degenerate training paths responsible for semantic drift and diversity loss.
• Fine-grained control: Domain-specific anchoring enables targeted influence over model capabilities, with robust prevention of out-of-domain drift.

Practically, this framework points to scalable and economically tractable self-evolving LLMs. From a theoretical viewpoint, it highlights the necessity of regularization even in closed-loop, reward-shaping self-improvement. It also underscores the role of curriculum and knowledge locality in maximizing competence acquisition.

Future Directions

Potential research avenues include:

• Extending guided self-evolution to domains where correctness lacks a deterministic verifier, necessitating softer or adversarial validation signals.
• Developing more sophisticated grounding and regularization (e.g., semantic, causal, multi-modal anchoring).
• Exploring larger sampling pools and dynamic anchor selection strategies for improved scalability and robustness.

Conclusion

The R-Few framework substantiates that stable and effective self-evolution of LLMs is possible with drastically reduced human supervision. Leveraging a tightly-coupled Challenger–Solver design, in-context human anchors, and online curriculum curation, it outperforms prior self-play methods in both stability and data efficiency. These findings reinforce the paradigm that human-guided but not human-dominated self-play is a viable and scalable approach for next-generation LLM reasoning, with both practical and theoretical ramifications for broader AI self-improvement strategies.