Collective Skill Reinforcement Learning

• Collective Skill RL is a paradigm where diverse agents share, transfer, and coordinate skills to solve tasks more efficiently than isolated learners.
• It employs mechanisms like distillation, latent variable exploration, and adaptive feedback to foster skill evolution and cross-domain transfer.
• The framework has applications in robotics, multi-agent systems, and AI coaching, showing empirical gains in efficiency and policy robustness.

Collective Skill Reinforcement Learning denotes a family of reinforcement-learning formulations in which reusable skills are acquired, transferred, selected, validated, or coordinated at the level of a population rather than an isolated learner. Across the literature, the “collective” can be a set of heterogeneous agents exchanging actionable knowledge across independent environments, a cooperative team selecting complementary latent skills, a skill bank of textual procedures for agentic LLMs, a federated robot swarm sharing policy parameters under bandwidth limits, or a human–AI dyad in which the AI coach optimizes the learner’s independent competence rather than short-term joint performance (Lin et al., 2017, Yang et al., 2019, Lin et al., 15 Jun 2026, Zhang et al., 7 Jun 2026, Na et al., 2022, Wang et al., 24 Jun 2026). The common thread is that skill formation is not treated as a purely individual optimization problem: information produced by one learner, role, model, or skill instance is used to shape the training signal, exploration space, or policy constraints of others.

1. Conceptual scope and distinctions

In the literature, Collective Skill Reinforcement Learning is not a single algorithmic template but a recurrent design principle. In one line of work, collective skill arises from cross-task knowledge exchange: Collaborative Deep Reinforcement Learning (CDRL) defines $m$ independent environments $\{\varepsilon_i\}$ and agents $\{g_i\}$, where actions taken by one agent do not affect other agents’ states, and collaboration occurs through information exchange rather than joint control (Lin et al., 2017). In another line, collective skill is explicitly multi-agent: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery learns complementary latent skill variables at a high level and primitive control at a low level, while execution remains decentralized (Yang et al., 2019). A third line treats the collective as a skill repository or skill tree for LLM agents, where multiple candidate procedures are generated, judged, compared, and reinforced during training (Lin et al., 15 Jun 2026, Zhang et al., 7 Jun 2026, He et al., 1 Jun 2026).

This breadth matters because several nearby concepts are distinct. Collective Skill RL is not synonymous with cooperative multi-agent RL. CDRL explicitly contrasts its framework with classical cooperative MARL: its objective is improved individual return via transfer across independent environments, not joint action in a shared environment (Lin et al., 2017). Nor is it equivalent to hierarchical RL in the narrow options sense. CSRL in "OpenClaw-Skill" conditions a shared policy on multiple skills-as-context and computes group-relative advantages across skill-conditioned rollouts, but does not require an explicit learned policy over skills (Lin et al., 15 Jun 2026). Likewise, SAPO and ReSkill treat skills as structured natural-language guidance conditioned on retrieval or triggers, rather than as conventional option policies with initiation and termination functions (Zhang et al., 7 Jun 2026, He et al., 1 Jun 2026).

The surveyed work therefore supports an umbrella interpretation. This suggests that Collective Skill RL is best understood as a class of mechanisms for collective acquisition and reuse of procedural knowledge under reinforcement learning, with the “collective” instantiated as peer agents, latent roles, skill banks, model ensembles, or coach–learner interactions.

2. Formal representations of skills and collective objectives

The notion of “skill” varies sharply across formulations. In CDRL, skills are encoded in action-logit distributions. A teacher agent $\alpha$ produces logits $z_t^\alpha$, which are aligned to a student task by a deep alignment network $g_\phi$, yielding aligned soft targets $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$. The student acts with its policy head $q_t=\mathrm{softmax}(z_t^\beta)$ but learns transferred knowledge through a separate distillation head $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$, optimized by

$$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$

This term is added to the actor–critic objective,

$\{\varepsilon_i\}$0

so that collective skill transfer enters as an auxiliary but on-line learning signal (Lin et al., 2017).

In hierarchical cooperative MARL, a skill is a latent variable $\{\varepsilon_i\}$1 sustained for $\{\varepsilon_i\}$2 low-level steps. Each agent has a high-level policy $\{\varepsilon_i\}$3 and a low-level policy $\{\varepsilon_i\}$4. Skill discovery is driven by a decoder $\{\varepsilon_i\}$5, and the low-level reward mixes extrinsic team reward with an intrinsic decodability term,

$\{\varepsilon_i\}$6

where $\{\varepsilon_i\}$7 (Yang et al., 2019). Here, collective skill is literally a coordinated distribution over temporally extended latent behaviors.

In LLM-centered systems, skills are textual procedures or contextual control variables. CSTS decomposes a task $\{\varepsilon_i\}$8, synthesizes candidate skill nodes $\{\varepsilon_i\}$9 from trajectories $\{g_i\}$0, and scores them by collective quality and collective transferability: $\{g_i\}$1

$\{g_i\}$2

CSRL then samples $\{g_i\}$3 rollouts for every skill in the candidate set $\{g_i\}$4 and computes cross-skill advantages

$\{g_i\}$5

followed by a clipped GRPO-style objective over the whole multi-skill batch (Lin et al., 15 Jun 2026). In SAPO, the key quantity is not cross-skill competition but marginal utility under matched conditions: $\{g_i\}$6 This turns collective skill management into a causal attribution problem over co-retrieved skills (Zhang et al., 7 Jun 2026).

Another formalization appears in AI coaching. The learner and coach form a two-player partially observable stochastic game with latent skill $\{g_i\}$7, and the executed action is a shared-control blend,

$\{g_i\}$8

The learner optimizes task performance, whereas the coach optimizes the learner’s future independent competence, using a surrogate reward $\{g_i\}$9 during training (Wang et al., 24 Jun 2026). This extends collective skill formation beyond machine populations to human–AI skill development.

3. Mechanisms of collective skill formation

A first mechanism is transfer under heterogeneity. CDRL addresses mismatched action spaces and action semantics with a deep alignment network $\alpha$0 and a dedicated distillation head, so that transferred supervision does not directly overwrite the student’s policy head. It supports both offline alignment, using trained teachers on both tasks, and online alignment, where the student first learns from its own environment and distillation is phased in later through thresholds $\alpha$1 and $\alpha$2 (Lin et al., 2017). ASPiRe uses a different but related strategy: it learns a library of specialized skill priors $\alpha$3, then an Adaptive Weight Module $\alpha$4 regularizes a downstream policy with a weighted sum of KL divergences,

$\alpha$5

This yields concurrent composition when multiple weights are active, and sequential composition when weight mass shifts over time (Xu et al., 2022).

A second mechanism is collective evaluation before adoption. CSTS uses multiple models both to generate candidate skill nodes and to judge them, explicitly separating collective quality from collective transferability (Lin et al., 15 Jun 2026). SAPO pushes this further with matched rollout design: given a task $\alpha$6 and retrieved skill set $\alpha$7, it splits the rollout budget into base rollouts conditioned on $\alpha$8 and skill-augmented rollouts conditioned on $\alpha$9, then promotes only candidates satisfying

$z_t^\alpha$0

The point is not merely to store skills, but to isolate the context-dependent marginal contribution of a candidate before it enters the collective (Zhang et al., 7 Jun 2026).

A third mechanism is RL-in-the-loop skill evolution. ReSkill embeds skill creation, testing, selection, and pruning inside GRPO training. New and old versions of a skill are assigned within a rollout group by Thompson Sampling with adaptive discounting,

$z_t^\alpha$1

and a new version is accepted only if $z_t^\alpha$2 under the discounted Beta posteriors (He et al., 1 Jun 2026). This prevents the auto-accept failure mode documented in the same paper. RLCCF uses a related collective-feedback mechanism at the model level: multiple LLMs produce $z_t^\alpha$3 samples each, each model receives a self-consistency score

$z_t^\alpha$4

and the collective pseudo-label is selected by SC-weighted voting,

$z_t^\alpha$5

Each model is then updated by a GRPO-style objective toward agreement with this collective consensus (Yuan et al., 17 Aug 2025).

A fourth mechanism is direct suggestion or teaching. Suggestion Sharing defines $z_t^\alpha$6 as an action-distribution recommendation from agent $z_t^\alpha$7 to agent $z_t^\alpha$8, and optimizes a dual-clipped objective with discrepancy penalties between suggestions and executed policies (Jin et al., 2024). Mutual Reinforcement Learning treats reward channels themselves as adaptive skills in a robot–human dyad: the expert robot samples reinforcers according to a learned weight vector over reward channels, updates the weights after success or failure, and tracks entropy $z_t^\alpha$9 as an information-gain proxy (Roy et al., 2019). AI coaching replaces symbolic reward channels with adaptive shared control, but the pedagogical logic is similar: effective assistance must be aligned with the learner’s capability and must strategically step back to enable productive failures (Wang et al., 24 Jun 2026).

4. Algorithmic families and representative systems

The major families of Collective Skill RL differ less by domain than by what is shared, aligned, or jointly optimized.

Family	Representative mechanism	Representative papers
Cross-task transfer	Logit alignment, distillation head, asynchronous actor–critic updates	CDRL (Lin et al., 2017)
Hierarchical team coordination	Latent skills, decoder-based intrinsic reward, centralized high-level coordination with decentralized execution	HSD (Yang et al., 2019), HRCL (Qin et al., 22 Sep 2025)
Skill-bank and skill-tree RL	Collective node generation/assessment, matched-rollout validation, RL-in-the-loop skill revision, adaptive prior weighting	CSTS+CSRL (Lin et al., 15 Jun 2026), SAPO (Zhang et al., 7 Jun 2026), ReSkill (He et al., 1 Jun 2026), ASPiRe (Xu et al., 2022)
Collective feedback and suggestion exchange	SC-weighted voting, collective consistency, action suggestions to peers	RLCCF (Yuan et al., 17 Aug 2025), Suggestion Sharing (Jin et al., 2024)
Federated and distributed robotics	Parameter aggregation, asynchronous replay-based policy search, OOD state initialization	FLDDPG (Na et al., 2022), ADGPS (Yahya et al., 2016), craft-robot CTDE with OODSI (Zhao et al., 24 Feb 2026)
Human–AI collective skill formation	Adaptive shared control, latent skill dynamics, pedagogical RL	Mutual RL (Roy et al., 2019), AI Coaching (Wang et al., 24 Jun 2026)

Within this taxonomy, two broad structural oppositions recur. One is centralized training with decentralized execution versus decentralized or federated learning. HRCL uses CTDE PPO at the high level together with decentralized EPOS plan selection at the low level, whereas FLDDPG and ADGPS retain decentralized data collection and use parameter averaging or shared replay to extract a collective policy (Qin et al., 22 Sep 2025, Na et al., 2022, Yahya et al., 2016). The other is explicit skill selection versus implicit conditioning. HSD and HRCL explicitly choose latent skills or high-level strategy groups, while CSRL, SAPO, and ReSkill often condition a shared policy on skills-as-context and let relative reward or marginal utility determine which procedures persist (Yang et al., 2019, Qin et al., 22 Sep 2025, Lin et al., 15 Jun 2026, Zhang et al., 7 Jun 2026, He et al., 1 Jun 2026).

A further distinction concerns whether the collective is homogeneous. RLCCF deliberately builds a heterogeneous ensemble—Qwen2.5-7B, GLM-4-9B, InternLM3-8B-Instruct, and LLaMA-3.1-8B-Instruct—to exploit complementary output distributions (Yuan et al., 17 Aug 2025). By contrast, HSD assumes homogeneous agents with shared observation and action spaces (Yang et al., 2019), while FLDDPG allows individualized local actor–critic networks but relies on equal-weight FedAvg plus soft local retention $g_\phi$0 to keep a coherent swarm-level navigation skill (Na et al., 2022).

5. Empirical findings

The reported results show that collective skill mechanisms can improve both learning efficiency and asymptotic performance, but the gains depend strongly on the substrate. In heterogeneous Atari transfer, CDRL reports that collaborative training from scratch raises Bowling performance from an A3C baseline of $g_\phi$1 to $g_\phi$2, a relative improvement of approximately $g_\phi$3 (Lin et al., 2017). In swarm robotics, FLDDPG achieves a simulation success rate $g_\phi$4 of $g_\phi$5, compared with $g_\phi$6 for IDDPG, $g_\phi$7 for SEDDPG, and $g_\phi$8 for SNDDPG, while also achieving the shortest completion time; in real-robot transfer it attains the highest success rate and lowest completion time among the compared strategies (Na et al., 2022). Distributed asynchronous guided policy search learns a single torque-level door-opening policy with per-robot success rates of $g_\phi$9, $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$0, $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$1, and $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$2, for a mean of $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$3 across four robots on a held-out test door (Yahya et al., 2016). ReinforceGen, which jointly improves initiation prediction, skill execution, and termination logic over a sequence of localized manipulation skills, reports overall visuomotor success of $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$4 versus $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$5 for HSP, described as an $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$6 average relative performance increase (Zhou et al., 18 Dec 2025).

On multi-agent coordination benchmarks, HSD demonstrates distinct and interpretable skills in a stochastic team sports game, with ad-hoc cooperation results that remain strong when paired with scripted teammates, unlike QMIX and IQL (Yang et al., 2019). HRCL reports substantial reductions in combined cost in synthetic and smart-city settings, including a $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$7 reduction versus EPOS-P in the basic synthetic scenario and $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$8 lower mean discomfort plus $$\tilde p_t=\mathrm{softmax}(g_\phi(z_t^\alpha)/\tau)$$9 lower inefficiency cost than MAPPO (Qin et al., 22 Sep 2025). Decentralized meta-RL on open-ended task trees achieves over $q_t=\mathrm{softmax}(z_t^\beta)$0 completion of training task trees and generalizes to task trees of twice the training depth, with a novel “pressure plate” task solved in $q_t=\mathrm{softmax}(z_t^\beta)$1 of trials despite zero exposure during training (Bornemann et al., 2023). In real-world craft robots, OODSI improves sim-to-real performance by $q_t=\mathrm{softmax}(z_t^\beta)$2 in the abstract, and in Gazebo evaluation PPO+DR+OODSI reaches $q_t=\mathrm{softmax}(z_t^\beta)$3 on the cooperative task and $q_t=\mathrm{softmax}(z_t^\beta)$4 on the competitive task (Zhao et al., 24 Feb 2026).

The most rapid recent expansion of Collective Skill RL is in agentic LLMs. OpenClaw-Skill reports an ablation on Qwen3.5-9B in which Base $q_t=\mathrm{softmax}(z_t^\beta)$5, $q_t=\mathrm{softmax}(z_t^\beta)$6CSN-Gen $q_t=\mathrm{softmax}(z_t^\beta)$7, $q_t=\mathrm{softmax}(z_t^\beta)$8CSN-Assess $q_t=\mathrm{softmax}(z_t^\beta)$9, and $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$0CSRL $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$1, isolating a further $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$2 contribution from CSRL beyond CSTS-only training (Lin et al., 15 Jun 2026). SAPO reaches an ALFWorld All score of $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$3, WebShop Score $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$4 and Success $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$5, and average Search-QA $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$6, while ablations show that removing validation reduces ALFWorld All from $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$7 to $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$8 and WebShop Success from $\tilde q_t=\mathrm{softmax}(z_t^{\beta'}/\tau)$9 to $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$0 (Zhang et al., 7 Jun 2026). ReSkill reports $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$1 on ALFWorld with Qwen3-4B, surpassing GRPO at $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$2 and SkillRL at $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$3, and also reports larger gains on unseen tasks and harder out-of-domain settings (He et al., 1 Jun 2026). RLCCF reports an average relative improvement of $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$4 in accuracy across four open-source LLMs and a $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$5 increase in group majority-voting accuracy, from $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$6 to $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$7 (Yuan et al., 17 Aug 2025). AT-GRPO reports especially large gains in long-horizon planning, increasing accuracy from a $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$8 to $$L_{\mathrm{distill}}=\mathbb{E}_t\!\left[\tau^2\,\mathrm{KL}(\tilde p_t\|\tilde q_t)\right].$$9 percent single-agent RL baseline to $\{\varepsilon_i\}$00 to $\{\varepsilon_i\}$01 percent, while also improving coding by $\{\varepsilon_i\}$02 to $\{\varepsilon_i\}$03 percent and math by $\{\varepsilon_i\}$04 to $\{\varepsilon_i\}$05 percent (Zhao et al., 13 Oct 2025).

Human-oriented variants show that collective skill objectives need not optimize only machine populations. In Baxter block building, Mutual Reinforcement Learning improves post-training performance relative to random feedback with $\{\varepsilon_i\}$06, and the learned group shows fewer mistakes in later Simpson categories than the other groups (Roy et al., 2019). In AI Coaching for drone racing, the L2C condition yields a lap-time reduction of $\{\varepsilon_i\}$07 with $\{\varepsilon_i\}$08 and failure reduction of $\{\varepsilon_i\}$09 per lap with $\{\varepsilon_i\}$10, whereas the compared baselines do not achieve significance on both objective measures (Wang et al., 24 Jun 2026).

6. Limitations, misconceptions, and future directions

A persistent misconception is that collective skill methods are simply cooperative MARL with another name. The literature does not support that simplification. CDRL is explicitly about information exchange across independent environments rather than shared-environment control (Lin et al., 2017). HRCL mixes high-level MARL with decentralized collective learning rather than replacing one with the other (Qin et al., 22 Sep 2025). AT-GRPO shows that role-conditioned prompting alone is insufficient: on-policy RL for collaborative LLMs requires agent- and turn-wise grouping because standard GRPO assumptions break down when prompts vary by role and by turn (Zhao et al., 13 Oct 2025).

The dominant technical risk is negative transfer or harmful collective interference. CDRL reports that naïve KD-policy hurts performance due to action-semantic mismatch and notes sensitivity to $\{\varepsilon_i\}$11, $\{\varepsilon_i\}$12, and task relatedness (Lin et al., 2017). CSTS and CSRL depend on judge diversity, verifier design, skill-tree quality, and decomposition coverage, while their transferability checks and multi-skill rollouts introduce computational overhead of $\{\varepsilon_i\}$13 and $\{\varepsilon_i\}$14 per subtask in the reconstructed accounting (Lin et al., 15 Jun 2026). SAPO notes that utility estimation depends on rollout samples and that non-stationarity in the policy causes the utility of prior skills to shift over time; it also lacks formal hypothesis testing or confidence intervals (Zhang et al., 7 Jun 2026). ReSkill documents that auto-accepting all skills during RL collapses performance mid-training, which is precisely why within-group testing and discounted Thompson Sampling are required (He et al., 1 Jun 2026).

Collective feedback itself can fail. RLCCF acknowledges wrong consensus if many models share similar biases, echo-chamber effects as the ensemble homogenizes, and sensitivity to self-consistency miscalibration (Yuan et al., 17 Aug 2025). Suggestion Sharing reduces privacy leakage relative to reward, value, or policy sharing, but still assumes truthful suggestions and scales with per-recipient suggestion heads (Jin et al., 2024). In human–AI coaching, the latent skill variable is observed only during training, deployment relies on Bayesian belief updates from performance proxies, and the probabilistic finite-state skill model is necessarily a coarse approximation of human motor learning (Wang et al., 24 Jun 2026).

The stated future directions converge on a few themes. Several papers call for stronger hierarchical structure: explicit high-level skill selection in CSRL, hierarchical skill libraries in ReSkill, and learned grouping criteria in HRCL (Lin et al., 15 Jun 2026, He et al., 1 Jun 2026, Qin et al., 22 Sep 2025). Others emphasize more robust collective estimation: median or trimmed-mean aggregation in CSTS, variance-aware estimators or Bayesian marginal-utility models in SAPO, and calibrated self-consistency weighting in RLCCF (Lin et al., 15 Jun 2026, Zhang et al., 7 Jun 2026, Yuan et al., 17 Aug 2025). Distributed systems work points toward decentralized critics, adaptive aggregation weighting, compression, and heterogeneous swarms (Qin et al., 22 Sep 2025, Na et al., 2022). Human-centered work points toward personalized skill-acquisition models, cohort-scale coaching, safety-aware stepping back, and multimodal coaching policies (Wang et al., 24 Jun 2026). This suggests that the next phase of Collective Skill RL will be defined less by whether skills exist in the system, and more by whether their creation, attribution, routing, and retirement can be made statistically reliable, scalable under heterogeneity, and aligned with long-horizon collective objectives.