Self-Learning Framework in LLM Agents

• A self-learning framework is a closed-loop system that enables autonomous agents to self-improve through iterative task generation, evaluation, and curriculum adjustments.
• The framework integrates co-evolving modules—Prompt Generator, Policy Model, and Generative Reward Model—to maintain reward integrity and prevent reward hacking.
• Empirical results demonstrate that scalable synthetic data and minimal human intervention drive continual performance enhancements in LLM-based agents.

A self-learning framework is a structured, often closed-loop system in which an agent—whether an artificial agent or a human learner—is endowed with mechanisms for iteratively expanding its knowledge, skills, or competencies without continuous external supervision or fixed data curation. Recent frameworks leverage co-evolving modules for task generation, solution, and evaluation, as well as architectural innovations that ensure adaptive curriculum, reward reliability, and continual improvement across complex environments and domains. This entry focuses on the formal, methodological, and empirical dimensions of self-learning frameworks, with particular emphasis on the Agentic Self-Learning (ASL) paradigm for LLM-based agents (Sun et al., 16 Oct 2025).

1. Fundamental Principles and Operational Structure

A self-learning framework is designed to enable sustained, autonomous improvement of an agent or model through iterative, internally coordinated processes. Core principles established in the ASL framework include:

• Closed-Loop Multi-Role Architectures: Essential components include a Prompt Generator (PG) that synthesizes tasks, a Policy Model (PM) that executes those tasks, and a Generative Reward Model (GRM) that evaluates outputs and supplies continuous reward signals. These roles, instantiated within a shared LLM backbone, interact in tightly coupled loops—each round forming a curriculum where difficulties escalate as solving proficiency rises.
• Reward Model Co-Evolution: Rather than relying on inflexible, rule-based evaluation signals (exact-match or substring-based), a generative, learned reward model (GRM) is trained in synchrony with the policy. This model is continually updated to match evolving solution distributions, enhancing reward integrity and avoiding policy collapse or reward hacking.
• Curriculum and Task Difficulty Control: Task generation is regulated by entropy-based signals from the GRM, incentivizing the PG to produce tasks that are neither trivial nor unsolvable, but instead maximize the spread (uncertainty) in PM performance.
• Synthetic Data Scaling: Unlimited on-the-fly generation of synthetic tasks allows the data distribution to naturally expand in step with policy capabilities, explicitly removing the ceiling imposed by static, human-labeled datasets.

The operational loop iterates through:

1. Training PG to propose tasks maximizing entropy across PM-GRM rollouts.
2. Training GRM on self-generated and progressively curated data, using RLVR to align with correctness references.
3. Training PM with policy gradients derived from GRM's binary verification signals.

2. Module Architectures and Interactions

Each module in ASL is defined by specific inputs, functional objectives, and inter-module dependencies:

• Prompt Generator (PG):
  • Inputs: Meta-prompts, prior queries/answers, and cumulative difficulty flags.
  • Function: Proposes candidate (question, answer) pairs, filtered by thresholded GRM scoring.
  • Curriculum Signal: Receives entropy reward r_{PG} = H(s_1,…,s_M) based on the distribution of success/failure in PM-GRM rollouts on a proposed task, driving the system toward tasks at the solvability threshold.
• Policy Model (PM):
  • Structure: LLM-based policy πθ(y|x) traversing > , <tool_call>, <answer> sequences. > - Action Space: Interleaves internal reasoning, external tool/information access, and answer composition. > - Reward: AWarded binary signal s from the current GRM on each solution trajectory. > > - Generative Reward Model (GRM): > - Role: Bernoulli classifier pφ(s|x,y); trained to match reference correctness (e.g., substring match), tolerant to near-synonyms. > - Training: RLVR (Reinforcement Learning from Verifiable Rewards) to maximize agreement with reference scoring, continually re-trained on new (x,y) pairs to track PM’s shifting distribution. > > Update Sequence: At iteration t: > > 1. Update PG via RL on reward r_{PG}. > > 2. Update GRM via RLVR on current verification data. > > 3. Update PM with policy gradient on task-response data. > Replay buffers accumulate data, ensuring temporal continuity and preventing catastrophic forgetting. > > ## 3. Learning Objectives and Training Dynamics > > The formal optimization objectives in ASL are as follows: > > - Policy Objective: > > > $$J(\theta) = E_{x\sim D_{PM},\,y\sim\pi_\theta(\cdot|x)}[r_{PM}(x,y)]$$ > > with gradient > > $$\nabla_\theta J(\theta) = E_{x,y}[\nabla_\theta \log \pi_\theta(y|x) \cdot r_{PM}(x,y)]$$ > > - GRM Objective: > > > $$J_{GRM}(\phi) = E_{x,y,s_{ref}}\left[E_{\hat{s}\sim p_\phi(\cdot|x,y)}[I(\hat{s}=s_{ref})]\right]$$ > > with gradient > > $$\nabla_\phi J_{GRM}(\phi) = E[\nabla_\phi \log p_\phi(\hat{s}|x,y)\,I(\hat{s}=s_{ref})]$$ > > - Joint Optimization: > > ASL alternates the three module updates, rather than jointly blending losses, to enable stable co-evolution. > > Reward Hacking and Mitigation: If GRM is not continually updated on-policy, PG discovers adversarial tasks that break the discriminator. Co-training GRM and periodic introduction of a small fraction (≈1%) of real, human-verified data re-anchor and lift the system's performance ceiling. > > ## 4. Synthetic Task Scaling, Data Flow, and Ablation Studies > > - Data Scaling Experiments: Empirical trials increasing synthetic task set sizes (1k, 10k, 46k) demonstrate monotonic gains in PM accuracy on held-out benchmarks. On-the-fly PG generation obviates the need for human-labeled expansion, enabling unbounded curriculum extension. > > - Early Iterations: Training is based exclusively on self-generated tasks; human verification is deferred. > > - Late-Stage Intervention: Inserting a minimal quota of externally-verified data into D_{GRM} yields further gains, particularly once co-evolution saturates. > > Role Co-evolution and Synergy: The three-metric analysis (PG difficulty, GRM verification sharpness, PM solving accuracy) evidences the coordinated, mutually reinforcing progression of the three ASL roles. > > Reward Hacking Ablation: Freezing GRM in the third iteration results in immediate entropy inflation and performance plateau, while continuous GRM adaptation preserves and extends improvement curves. > > Empirical Summary Table: > > > | GRM Setting | PM Accuracy (after 3 iters) | Observation | > |---------------------|-----------------------------|---------------------------| > | Frozen | 42.1% | Reward hacking, stall | > | Self-data trained | 48.3% | Continues improving | > | +1% real data | 51.7% | Ceiling lifted | > > ## 5. Empirical Outcomes and Comparative Benchmarks > > Comparisons to RLVR-based baselines (e.g., Search-R1), and RL-without-reward/zero-shot baselines (Absolute Zero, R-Zero), demonstrate ASL’s superior sustained per-round gains and robustness in zero human-labeled-data conditions. While strongest RLVR policies plateau or degrade after an initial phase, ASL maintains monotonic, curriculum-driven policy improvement (Sun et al., 16 Oct 2025). > > Key outcomes: > > - Sample efficiency: ASL surpasses baselines with less or no human data. > > - Reward model dynamism: Continual GRM adaption is necessary for sustained progress; otherwise, the system is vulnerable to reward hacking. > > - Data scaling: Synthetic QA data volume is a primary determinant of maximal policy capacity. > > - Final accuracy: Small late-stage real data lift test accuracy by several additional points. > > ## 6. Limitations, Bottlenecks, and Directions for Extension > > - GRM Bottleneck: Verification capacity of the GRM is the principal constraint; if its discrimination saturates, overall learning progress halts. The PG cannot escalate difficulty beyond the GRM's ability to judge, leading to recurring reward hacking. > > - Practical Remedy: Two-phase regime—co-train GRM across self-generated data, then inject high-quality real data for re-anchoring. > > - Extension: Expanding ASL to multi-modal tool environments, multi-turn interactive agents, and embodied real-world agents is anticipated as future research (Sun et al., 16 Oct 2025). > > ## Summary > > The Agentic Self-Learning framework establishes a robust, fully closed-loop system for LLM-based agent training in open-domain search environments. It demonstrates that joint, curriculum-driven co-evolution of task generator, solver, and reward model, underpinned by scalable synthetic data generation and continual policy-reward adaptation, enables self-improving agents that outperform conventional rule-driven RL approaches. The framework’s critical levers are the fidelity and adaptivity of the generative reward model, and the unlimited generation of training data, validated through empirical ablation studies and extensive benchmark comparisons (Sun et al., 16 Oct 2025).