SAGE: Skill Augmented GRPO Framework

• SAGE is a framework that integrates skill augmentation into GRPO, enabling agents to autonomously acquire, deploy, and evolve reusable strategies across diverse environments.
• It employs mechanisms like self-hinted rollouts, skill libraries, and tree-structured RL to overcome sparse reward challenges and enhance exploration.
• Empirical results demonstrate consistent gains in domains such as reasoning, embodied control, and tool-use, highlighting improved sample efficiency and generalization.

Skill Augmented GRPO for Self-Evolution (SAGE) frameworks extend Group Relative Policy Optimization (GRPO) with skill-augmentation mechanisms, enabling LLMs and multimodal agents to autonomously acquire, deploy, and evolve reusable strategies—skills—across diverse environments. By integrating skills into the reinforcement learning (RL) loop, SAGE systems systematically overcome the limitation of sparse or delayed rewards inherent to complex reasoning, embodied, and tool-using domains, and demonstrate robust, generalizable improvements in both in-distribution and out-of-distribution tasks (Liao et al., 3 Feb 2026, Tian et al., 26 Jun 2025, Wang et al., 18 Dec 2025, He et al., 1 Jun 2026, Liang et al., 21 May 2026, Tian et al., 30 Apr 2026).

1. Foundations: GRPO and the Challenges of Sparse Rewards

Group Relative Policy Optimization (GRPO) partitions RL rollouts into fixed-size task-specific groups and computes a group-normalized advantage for each trajectory: $A_i = \frac{R_i - \bar{R}}{s + \varepsilon}$ where $R_i$ is the verifier reward, $\bar{R}$ the group mean, $s$ the within-group standard deviation, and $\varepsilon$ a small stabilizer. The GRPO objective,

$$J_{\rm GRPO}(\theta) = \mathbb{E}_{x,\tau} [A_{\rm group}(x,\tau)]$$

is optimized using trajectory-level PPO-like surrogates. This approach aligns LLMs with verifiable objectives efficiently but fails under sparse reward regimes common in math, logic, and embodied environments, where most groups yield identical (often zero) rewards, collapsing advantage estimates and halting gradient updates (Liao et al., 3 Feb 2026).

2. Skill Augmentation Mechanisms Across Domains

SAGE generalizes by integrating skills—structured, reusable procedures or guidance—into the GRPO loop to diversify rollout outcomes and provide auxiliary learning signals.

• Self-hinted rollouts inject structured hints or plans at training-time to break degenerate reward groups, enhancing exploration diversity and restoring nonzero gradients (Liao et al., 3 Feb 2026).
• Skill libraries systematically acquire and deploy code-defined skills across sequential tasks, providing additional reward bonuses and stateful memory for compositional tool use (Wang et al., 18 Dec 2025).
• Tree-structured RL leverages Monte Carlo Tree Search (MCTS) to generate dense process rewards for sibling actions, feeding back per-node skill utility and supporting on-policy, tree-based advantage normalization (Tian et al., 26 Jun 2025).
• Assertion-driven skill creation automatically diagnoses failure clusters, generates targeted reusable skill modules with tailored triggers, and validates skill efficacy online through A/B testing (He et al., 1 Jun 2026).
• Duality or geometric operation pools enforce logical consistency across paired transformations (visual or linguistic), dynamically scheduling skill-related auxiliary signals in spatial reasoning domains (Liu et al., 18 May 2026).
• Training-free GRPO employs group-wise offline optimization of skill prompt/code variants, selecting the best-performing skill instance via comparative execution and regularization against distributional drift (Tian et al., 30 Apr 2026).

3. SAGE: Principled Training Objectives and Curriculum Design

All SAGE variants adapt GRPO with skill-conditioned or skill-augmented objectives, operating over either token or trajectory levels. Common SAGE modifications include:

• Skill-conditioned rollouts: At each RL step, the policy samples or activates a skill (or hint), executes the rollout, and computes reward/advantage conditioned on that skill’s presence (Liao et al., 3 Feb 2026, Wang et al., 18 Dec 2025).
• Skill-integrated rewards: In addition to base environment rewards, agents receive auxiliary rewards if skills are created, invoked, or transferred successfully between tasks. For example:

$$R_{\text{skill}} = r_{\text{base}} + \lambda \cdot \mathbf{1}_{\text{skill used}}$$

• Adaptive skill scheduling: Dynamic schemas select when and which skills (or skill strengths) to activate based on current performance plateaus, agent bottlenecks, or operation pool states. Policy-dependent scheduling, as in self-hinting, triggers increased hint levels only when learning stalls (Liao et al., 3 Feb 2026). Assertion-driven mechanisms in ReSkill similarly gate skill bank updates via explicit failure diagnosis and A/B selection (He et al., 1 Jun 2026).
• Offline skill evolution: Skills are mutated, tested, or distilled offline, decoupling optimization of exploration (policy learning) from exploitation (skill consolidation). This two-stage loop prevents destabilization due to premature skill adoption and enables robust policy-skill co-evolution (Liang et al., 21 May 2026, Tian et al., 30 Apr 2026).

4. Implementation Workflows and Algorithmic Variants

Several canonical pipelines and pseudocode structures emerge across SAGE implementations:

Variant	Key Skill Mechanism	Policy Update
Self-Hint SAGE	Privileged self-hints (plans, decomps.)	On-policy, w/ hints
ReSkill	Assertion-based, triggered skills	A/B test in-group
Tree-SAGE	Tree-MCTS + skill-planning	Tree-GRPO
Skill-Library SAGE	Sequential task chain, code skill reuse	Chained GRPO
Consistency-SAGE	Duality/geometric logic ops as skills	Consistency-aug.
Training-Free SAGE	Skill prompt/code mutation, no retrain	Groupwise selection

• Self-hint SAGE jointly optimizes policy and hint-generator via on-policy advantage, adaptively raising hint-level per prompt when all-group rewards collapse (Liao et al., 3 Feb 2026).
• ReSkill interleaves in-group A/B rollout assignment (old vs new skill bank), assertion-based diagnostics for skill proposal, and bandit allocation (Thompson Sampling with adaptive discounting) to balance exploration and exploitation (He et al., 1 Jun 2026).
• Tree-SAGE builds MCTS trees, computes sibling process rewards, and applies Tree-GRPO with per-token normalization for both policy and reward model updates (Tian et al., 26 Jun 2025).
• Skill-library SAGE accumulates and invokes code-defined skills across sequential task chains, assigning reward bonuses for library creation/use and optimizing the policy via chained GRPO updates (Wang et al., 18 Dec 2025).
• Consistency-SAGE maintains a dynamic pool of duality/geometric operations, performing consistent-check rollouts with transformed inputs, adjusting reward functions accordingly, and managing active/mastered/candidate operation states (Liu et al., 18 May 2026).
• Training-free SAGE (Skills-Coach) generates prompt/code variants for each skill, executes against a task suite, and selects the highest-performing variant per GRPO group, subject to code drift regularization (Tian et al., 30 Apr 2026).

5. Empirical Results and Domain Coverage

SAGE methods have been quantitatively validated across LLM reasoning, search-augmented QA, agentic tool-use, embodied RL, and vision-language spatial tasks.

Domain/Benchmark	SAGE Variant	Key Result/Delta	Reference
Math/Reasoning (AIME24…)	Self-hint SAGE	+2.0–1.3 accuracy vs GRPO	(Liao et al., 3 Feb 2026)
Search Aug. QA (QA-7)	Search-E1 (self-distil.)	+3.5 pp EM over RL baseline	(Liang et al., 21 May 2026)
Embodied (ALFWorld)	Tree-GRPO SAGE	85.2% avg vs 73.4–81.1% (baselines)	(Tian et al., 26 Jun 2025)
AppWorld/Tool-use	Skill-Library SAGE	+8.9pp scenario completion, -59% tokens	(Wang et al., 18 Dec 2025)
Spatial Video QA	Consistency-SAGE	+4.5–14.5 pts on video, +3–15 pts spatial	(Liu et al., 18 May 2026)
Skill-specific QA (Skill-X)	Training-Free SAGE	88% pass (from 33.6%), +0.462 average	(Tian et al., 30 Apr 2026)
Agentic RL, OOD Gen.	ReSkill	89.6% (ALFWorld), SOTA OOD gains	(He et al., 1 Jun 2026)

Across all settings, SAGE reliably improves not only average-case performance but also sample efficiency and out-of-distribution generalization, leveraging skill acquisition for robust, transferable credit assignment.

6. Analysis of Algorithmic Components and Ablations

Major ablation and analysis highlights include:

• Offline vs Online skill/hint evolution: Online (on-policy, self-updating) skill conditioning provides strongest improvement and adaptivity to changing policy states (Liao et al., 3 Feb 2026, He et al., 1 Jun 2026).
• Adaptive gating/scheduling: Dynamically raising skill/hint level or operation pool membership prevents wasted compute on already-mastered items and concentrates training signal on current failure modes (Liao et al., 3 Feb 2026, Liu et al., 18 May 2026).
• A/B testing and Thompson Sampling: Controlled skill vs base policy comparisons (within-group) are critical for rejecting non-beneficial skills, mitigating regression and catastrophic forgetting (He et al., 1 Jun 2026).
• Reward and advantage normalization: Group and token-level normalization stabilizes gradients in high-variance and sparse-reward settings, critical for scaling up batch/group sizes (Tian et al., 26 Jun 2025, He et al., 1 Jun 2026).
• Skill drift and regularization: Limiting update steps or enforcing explicit KL/divergence constraints on skill evolution is necessary to prevent over-specialization and performance collapse (Tian et al., 30 Apr 2026).

A plausible implication is that modularization of skill creation, gated deployment, and robust online validation are essential for sustainable policy-skill co-evolution, especially in environments with shifting reward structures or heterogeneous tasks.

7. Generalization, Limitations, and Future Directions

SAGE frameworks provide a unifying recipe for equipping LLM and agentic systems with autonomous self-evolution capacity through modular skill integration with GRPO. By leveraging groupwise advantage normalization, adaptive skill scheduling, and modular reward integration, SAGE methods overcome sparse credit assignment bottlenecks and scale robustly across complex environments.

However, common challenges remain: scalability to very large or unbounded skill banks; efficient retrieval and context selection under memory constraints; safe handling of adversarial or degenerate skill proposals; and balancing rapid adaptation with policy stability. Ongoing research explores hybrid approaches, such as interleaving world model-based simulation, incorporating cross-domain skill transfer, or integrating richer hierarchical retrieval for skill libraries.

Continued development of SAGE systems is expected to yield increasingly autonomous, sample-efficient, and generalizable agents capable of open-ended skill discovery, tool synthesis, and robust task adaptation in both simulated and real-world settings (Liao et al., 3 Feb 2026, He et al., 1 Jun 2026, Wang et al., 18 Dec 2025, Tian et al., 26 Jun 2025, Liang et al., 21 May 2026, Tian et al., 30 Apr 2026, Liu et al., 18 May 2026).