Group-Relative Skill Learning in AI

• Group-relative skill learning is an approach that evaluates skills relative to a comparison set, emphasizing transformational change over absolute outcomes.
• It employs methodologies like relative variational intrinsic control and group-normalized advantages to drive skill evolution in diverse settings.
• This framework reduces bias, prevents absolute state shortcuts, and promotes personalized skill curation in complex, heterogeneous environments.

to=arxiv_search.search 天天中彩票是不是json 五分彩 1 {"query":"(Baumli et al., 2020) Relative Variational Intrinsic Control"} to=arxiv_search.search 重庆时时彩彩ք 早点加盟 5 {"query":"Relative Variational Intrinsic Control (Baumli et al., 2020)"} to=arxiv.search գործում 大发游戏官网 5 {"query":"(Baumli et al., 2020)"} Group-relative skill learning denotes a family of methods in which skills are learned, valued, selected, revised, or composed relative to a structured comparison set rather than as absolute properties of isolated behaviors. In the cited literature, the relevant “group” may be the initial state relative to the final state, a set of peer agents, a rollout group under shared task conditions, a client population with heterogeneous capability boundaries, or a cohort whose members should be combined under fairness constraints. The common aim is to prevent absolute-state shortcuts, confounding by skill composition, or unstructured skill accumulation, and to replace them with objectives that measure relative transformation, marginal utility, complementarity, transferability, fairness, or personalized benefit (Baumli et al., 2020, Yang et al., 2019, Yang, 2018, Jenkins et al., 2023, Yang et al., 2 Jun 2026, He et al., 1 Jun 2026).

1. Conceptual scope and defining idea

In the cited work, group-relative skill learning is not a single canonical algorithm. It is instead a recurring design principle: skill-related signals are made meaningful by conditioning them on a reference group or relation. In unsupervised reinforcement learning, the reference is often the pair $(s_0,s_T)$ rather than $s_T$ alone, so that a skill is distinguished by how it changes the agent’s relation to the environment rather than by the absolute terminal state (Baumli et al., 2020). In hierarchical multi-agent reinforcement learning, the relevant group is the set of agents jointly selecting complementary latent skills, with skill quality measured through team reward and centralized coordination (Yang et al., 2019). In agentic LLM systems, the group may be a matched pair of rollout sets, a set of candidate skills for one subtask, a sequence of related tasks, or a population of heterogeneous clients whose skills evolve collaboratively but not identically (Zhang et al., 7 Jun 2026, Lin et al., 15 Jun 2026, Ouyang et al., 7 May 2026, Yang et al., 2 Jun 2026).

A central motivation across these works is that absolute or unconditional signals are often misleading. Mutual-information skill discovery can collapse to state-space partitioning when a discriminator can infer the skill from $s_T$ alone (Baumli et al., 2020). Move statistics can be confounded because conditioning on a move also conditions on stronger players, so $\mathbb{P}(W\mid M)$ need not equal the move’s intrinsic value (Yang, 2018). Global skill aggregation can be suboptimal when clients differ in backbone, harness, or task distribution, because a single uniform library averages away client-specific capability boundaries (Yang et al., 2 Jun 2026). This suggests that the defining contribution of group-relative formulations is not merely “using groups,” but identifying the comparison structure that isolates the intended notion of skill.

2. Formal patterns of relativity

The formal mechanisms used across the literature differ, but they share a common structure: compare performance or identifiability within a controlled group, then use the difference as the skill signal.

In Relative Variational Intrinsic Control, the skill variable $\Omega$ is trained to maximize discriminability given $(s_0,s_T)$ while minimizing discriminability given $s_T$ alone. With inverse predictors $q_\phi(\Omega\mid s_T,s_0)$ and $q_\psi^{\mathrm{abs}}(\Omega\mid s_T)$, the target quantity is

$$I(s_0,\Omega\mid s_T)=H(\Omega\mid s_T)-H(\Omega\mid s_T,s_0) \approx \mathbb{E}\big[\log q_\phi(\Omega\mid s_T,s_0)-\log q_\psi^{\mathrm{abs}}(\Omega\mid s_T)\big],$$

and the practical reward is

$s_T$0

The relative signal is therefore the advantage of relational predictability over absolute predictability (Baumli et al., 2020).

In group-based policy optimization for agentic LLMs, the relative signal is often a within-group baseline. ReSkill adopts GRPO’s group-normalized advantage

$s_T$1

then uses within-group rollout assignment to compare old and new skill versions under the same task and policy (He et al., 1 Jun 2026). Skill-R1 introduces a bi-level construction in which the intra-generation term compares rollouts sharing the same skill,

$s_T$2

while the inter-generation term compares successive skill revisions,

$s_T$3

so that a revision is rewarded not only for producing good rollouts now, but for improving the next generation’s rollout population (Vishe et al., 10 May 2026).

SAPO uses matched rollout groups under identical task and retrieval context. The candidate skill’s context-dependent marginal utility is estimated by the reward gap

$s_T$4

Operationally, base and skill-augmented rollouts differ only by inclusion of the candidate skill, and the resulting difference estimates the skill’s marginal contribution beyond already retrieved skills (Zhang et al., 7 Jun 2026).

Other settings instantiate the same logic with different mathematical objects. OpenClaw-Skill evaluates each candidate skill using collective quality and collective transferability,

$s_T$5

and selects by $s_T$6 (Lin et al., 15 Jun 2026). Fair student grouping learns latent skill coordinates $s_T$7 from Laplacian eigenmaps and then maximizes within-group dissimilarity $s_T$8 under size and fairness constraints (Jenkins et al., 2023). Skill-bias correction in games subtracts a group-relative skill baseline from naive conditional win rates, yielding the intrinsic move value

$s_T$9

Here, the “group” is the move-induced difference between player populations (Yang, 2018).

Setting	Group notion	Relative signal
RVIC	Initial–final state pair	$s_T$0
Skill-R1 / ReSkill / SAPO	Rollout group under shared context	Advantage or reward gap within matched groups
OpenClaw-Skill / FederatedSkill	Model or client population	Transferability or personalized aggregation
Student grouping / skill-bias correction	Cohort or move-conditioned population	Complementarity or debiased intrinsic value

3. Relative skill discovery in reinforcement learning and multi-agent systems

The most explicit early formulation appears in RVIC. VIC and DIAYN make each skill identifiable from produced states, but without constraining how skills differ, they often produce trivially diverse behaviors that partition absolute terminal states. RVIC addresses this by rewarding policies that are discriminable from $s_T$1 yet not from $s_T$2 alone, thereby forcing skills to carry information about the initial condition. The result is described as “tiling the space of affordances” rather than partitioning target states (Baumli et al., 2020).

Empirically, this relative formulation changes the learned behavior. In Reacher, VIC skills converge to distinct arm end-states irrespective of start, whereas RVIC skills produce consistent relative rotations across start configurations. In Atari, RVIC learns directional movement skills consistent across positions, whereas VIC largely partitions top-of-frame states and fails to generalize to unseen bottom regions. In hierarchical reinforcement learning over the first 40M learner steps, RVIC skills typically yield faster learning than VIC skills and a primitive-action-only baseline on most games tested (Baumli et al., 2020).

Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery extends the relative idea to coordinated teams. Low-level policies $s_T$3 learn decodable skills via an intrinsic reward based on the decoder probability $s_T$4, while high-level policies $s_T$5 choose complementary latent skill variables under centralized training with QMIX. The group-relative element is the joint high-level choice $s_T$6, whose value is learned from team reward and factored for decentralized execution (Yang et al., 2019). The resulting skills become interpretable as offense or defense roles, and the paper reports robust ad-hoc cooperation when teammates are scripted or fixed to offensive or defensive skills (Yang et al., 2019).

Peer Learning offers a different multi-agent formulation. Agents train in separate but equivalent MDPs, exchange only the advisee’s current state and action recommendations, and select which peer to trust via Boltzmann sampling over trust weights. Trust Values and Agent Values are updated from outcomes of following peer advice, and the advantage-style update

$s_T$7

measures whether peer $s_T$8 improved on the advisee’s own baseline (Derstroff et al., 2023). This makes peer reliability explicitly group-relative: advice is valuable only insofar as it improves performance relative to the current agent.

4. Group-relative skill evolution in agentic LLMs

Recent work on agentic LLMs generalizes group-relative skill learning from behavior discovery to skill evolution, validation, and curation. A recurrent theme is that skills should not be stored or adopted on the basis of isolated rollouts or global averages.

SAPO formalizes pre-storage skill validation by splitting the standard rollout budget into matched base and skill-augmented groups under the same query and retrieved context. The cross-group reward gap estimates the candidate skill’s context-dependent marginal utility before the skill enters the long-term bank. Skills are promoted only if $s_T$9, belong to the top $\mathbb{P}(W\mid M)$0 fraction in the temporary bank, and satisfy a novelty threshold against the existing bank (Zhang et al., 7 Jun 2026). The same marginal-utility signal is then used to train the policy as a skill generator and scorer.

Skill-R1 shifts the optimization target from the task LLM to a trainable skill generator that conditions a frozen executor. Its bi-level group-relative objective couples within-generation rollout comparison to inter-generation improvement, so that a useful revision must both steer the current frozen task model well and improve the next round of skill proposals. On GAIA, the GRPO-trained version reaches 41.8% overall accuracy, compared with 29.7% for Vanilla GRPO and 6.1% for the no-skills baseline; on WebWalker, the corresponding numbers are 26.0%, 22.0%, and 2.0% (Vishe et al., 10 May 2026).

ReSkill exploits GRPO’s group rollouts as controlled A/B tests for skill versions. Each rollout in a group is assigned either $\mathbb{P}(W\mid M)$1 or $\mathbb{P}(W\mid M)$2 by Thompson Sampling, while all rollouts share the same task and policy. Beta-Bernoulli posteriors with adaptive discounting update evidence for each version, and a candidate is accepted if $\mathbb{P}(W\mid M)$3 at the end of the evaluation cycle (He et al., 1 Jun 2026). The paper reports 89.8% overall on ALFWorld for the 4B setting versus 75.3% for GRPO and 83.9% for SkillRL, with the largest gains on unseen splits (He et al., 1 Jun 2026).

SkillOS moves the group-relative structure from rollout groups to grouped task streams. Each group begins with an empty repository, so the first task acts as a within-group baseline; later tasks test whether curation actions improved the frozen executor on related tasks. The curator is optimized with a composite reward

$\mathbb{P}(W\mid M)$4

and a group-relative advantage across multiple rollouts of the same task group (Ouyang et al., 7 May 2026). Repositories evolve from narrow skills toward more richly structured Markdown files encoding higher-level meta-skills (Ouyang et al., 7 May 2026).

OpenClaw-Skill and FederatedSkill extend the group notion further. OpenClaw-Skill uses heterogeneous models both to generate candidate skills and to judge them on quality and cross-model transferability, then applies Collective Skill Reinforcement Learning with cross-skill normalized advantages (Lin et al., 15 Jun 2026). FederatedSkill makes the group a set of heterogeneous clients. Clients exchange semantic skill diffs rather than raw trajectories, and the server-side evolution agent aggregates these patches into strictly personalized updates using client profiles, verified reward, capability boundaries, and conservative admission rules. Personalized evolution outperforms the “Global” non-personalized baseline by +9.8 to +12.2 percentage points across clients (Yang et al., 2 Jun 2026).

5. Group-relative composition, fairness, and debiasing

Not all group-relative skill learning concerns control policies. In student group formation, the objective is to compose groups whose members are complementary in a learned latent skill space while preserving fairness with respect to sensitive attributes. Course-mark vectors are embedded with Laplacian eigenmaps into spectral coordinates $\mathbb{P}(W\mid M)$5, and group diversity is measured by pairwise distances $\mathbb{P}(W\mid M)$6. The resulting constrained graph partitioning problem maximizes within-group dissimilarity subject to clique consistency, group size bounds, and lower bounds on the balance metric for each sensitive attribute (Jenkins et al., 2023). In the 10-student demonstration, max-diversity plus fairness achieved $\mathbb{P}(W\mid M)$7, whereas max-diversity without fairness produced $\mathbb{P}(W\mid M)$8 and $\mathbb{P}(W\mid M)$9 (Jenkins et al., 2023).

“Removing Skill Bias from Gaming Statistics” supplies a complementary perspective: skill learning can also mean learning intrinsic action values that are invariant to the skill composition of the data-generating group. Conditioning on a move selects stronger players when stronger players disproportionately make that move, so naive conditional win rates conflate move quality with player quality. The paper’s correction subtracts a cross-group skill baseline and rescales by a factor involving $\Omega$0, yielding an intrinsic value that is invariant to whether the data comes from a strong or weak player pool (Yang, 2018). This directly challenges the common assumption that “learning from the experts” reveals intrinsic move quality.

Taken together, these works show that group-relative skill learning can refer to at least three distinct operations: learning skills that are relational rather than absolute, composing groups relative to learned skill complementarity and fairness constraints, and removing distortions caused by group-level confounding when estimating skill or move value. The underlying comparison structure differs, but the methodological logic is the same.

6. Limitations, misconceptions, and open directions

A recurring misconception is that any diversity-promoting skill objective yields reusable skills. RVIC explicitly argues that this is false: if a skill is predictable from $\Omega$1 alone, the policy may collapse to absolute goals and state-space partitioning. Relative conditioning alleviates this, but the paper also notes limitations such as a fixed discrete uniform skill prior, fixed skill durations, sensitivity to explicit feature design, and reliance on visual encoders in pixel-only settings (Baumli et al., 2020).

A second misconception is that more collaborative data automatically yields better shared skills. FederatedSkill reports strong empirical privacy with semantic patches and low leakage audits, but it does not use formal differential privacy or secure aggregation, and adversarial clients and poisoned patches are out of scope (Yang et al., 2 Jun 2026). OpenClaw-Skill similarly benefits from collective generation and assessment, yet remains sensitive to ensemble composition and incurs additional cost from cross-model transferability checks (Lin et al., 15 Jun 2026).

A third misconception is that skill banks should grow whenever a candidate skill appears promising. SAPO is motivated precisely by the failure of store-first pipelines, arguing that delayed feedback obscures the marginal contribution of any individual skill once multiple retrieved skills interact (Zhang et al., 7 Jun 2026). ReSkill’s auto-accept ablation shows that untested skill injection can damage policy learning, while SkillOS shows that grouped, long-horizon curation signals are needed to learn useful update and delete behavior rather than insert-only accumulation (He et al., 1 Jun 2026, Ouyang et al., 7 May 2026).

Finally, several works identify common structural limits. Skill-R1 depends on verifiable rewards and is therefore less directly applicable when no reliable verifier exists (Vishe et al., 10 May 2026). HSD learns first-order individual skills rather than higher-order joint skills and assumes synchronized option boundaries (Yang et al., 2019). Student grouping remains computationally burdened by the edge-based clique formulation and by the use of marks as skill proxies (Jenkins et al., 2023). Skill-bias correction assumes that skill is the primary confounder and that per-move effects remain small enough for the linearized derivation to hold (Yang, 2018).

These limitations suggest that the field’s open problems are less about inventing ever more “skills” and more about specifying the correct relational baseline. Across reinforcement learning, agentic LLMs, federated evolution, and constrained group formation, the decisive question is which comparison group makes a skill signal meaningful for the downstream use case.