RPG: Relation Patterns in Continual MARL

• The paper introduces RPG, a continual multi-agent reinforcement learning method that decouples task-agnostic relation patterns from task-specific decision-making using a conditional hypernetwork.
• It employs a scalable relation capturer with sparse attention to extract dynamic features from evolving observation and action spaces, thereby addressing catastrophic forgetting.
• Empirical results on SMAC and LBF demonstrate that RPG achieves superior plasticity and efficiency, using fewer parameters compared to baselines like MACPro and EWC.

Searching arXiv for the specified paper to ground the article in the published record. General Relation Patterns-Guided Task-Specific Decision-Maker (RPG) is a continual multi-agent reinforcement learning method for cooperative settings in which observation spaces, action spaces, and entity sets change across tasks. Introduced in "From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination" (Yao et al., 8 Jul 2025), RPG separates two components of continual coordination: task-agnostic relation patterns, which encode how entities influence an agent’s decisions, and task-specific decision-making, which maps those patterns into action spaces that may vary from task to task. The method combines a scalable relation capturer, a conditional hypernetwork that generates task-specific decision-makers, and regularization terms on both modules to mitigate catastrophic forgetting while retaining plasticity for new tasks.

1. Formal setting in continual multi-agent coordination

RPG is formulated for Continual Multi-Agent Reinforcement Learning (Co-MARL), where tasks arrive sequentially and cooperative policies must be adapted without severe degradation on previously learned tasks. The paper models this setting as a Continual Decentralized Partially Observable Markov Decision Process (Co-Dec-POMDP). For each task $c$, the task tuple is

$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$

where $\mathcal{M}_c$ is the entity set, $\mathcal{N}_c \subseteq \mathcal{M}_c$ the agents, $s_t \in \mathcal{S}_c$ the global state, $o_t^i \in \mathcal{O}_c$ the local observation of agent $i$, and $a_t^i \in \mathcal{A}_c$ its action. The joint action $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$ induces transition dynamics $\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$ and shared reward $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$0. The optimization target is

$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$1

The continual aspect is central: when a new task $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$2 arrives, $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$3 changes to $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$4, which in turn changes the dimensionality of $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$5, $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$6, and $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$7. RPG adopts centralized training with decentralized execution (CTDE): each agent learns local action-values $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$8, while a mixing network combines these into $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$9 during training; only local policies are executed at test time (Yao et al., 8 Jul 2025).

This formulation places RPG within value-factorized cooperative MARL, but the problem definition differs from static-task MARL because the relevant challenge is not only coordination under partial observability, but coordination under sequentially changing entity cardinalities and action vocabularies. In that setting, catastrophic forgetting becomes coupled to representational mismatch: an agent must preserve generally useful coordination structure even as the interface between that structure and task-specific action spaces changes.

2. Relation patterns as the core representation

RPG identifies the core of continual multi-agent coordination as relation patterns: general, task-agnostic understandings of how entities affect an agent’s decisions. The paper motivates this using examples such as similar “focus-fire” interactions across multiple StarCraft scenarios, while noting that similar interaction structures can yield different returns when mapped into different action spaces. This separation between general interaction structure and task-specific action realization is the conceptual basis of the method (Yao et al., 8 Jul 2025).

The relation capturer operates on dynamic observation spaces by decomposing agent $\mathcal{M}_c$0’s local observation $\mathcal{M}_c$1 into self information $\mathcal{M}_c$2, teammate information $\mathcal{M}_c$3, and other-entity information $\mathcal{M}_c$4. A permutation-invariant cross-attention mechanism uses the self representation as query $\mathcal{M}_c$5 and teammate or other-entity representations as keys $\mathcal{M}_c$6 and values $\mathcal{M}_c$7. The attention weights are

$\mathcal{M}_c$8

These are used to compute weighted relational summaries

$\mathcal{M}_c$9

which are concatenated with encoded self features $\mathcal{N}_c \subseteq \mathcal{M}_c$0 to form the agent-level relation pattern

$\mathcal{N}_c \subseteq \mathcal{M}_c$1

A GRU then accumulates relational history over time,

$\mathcal{N}_c \subseteq \mathcal{M}_c$2

and the resulting hidden state serves as the output of the relation capturer. Functionally,

$\mathcal{N}_c \subseteq \mathcal{M}_c$3

with $\mathcal{N}_c \subseteq \mathcal{M}_c$4 denoting capturer parameters.

The capturer is regularized by an entropy term that encourages sparse attention: $\mathcal{N}_c \subseteq \mathcal{M}_c$5 The intended effect is to concentrate relational computation on key entities rather than distributing attention uniformly across all visible entities. In the paper’s interpretation, this supports scalability in dynamic entity settings and improves decision quality by emphasizing interactionally salient objects.

The paper’s analysis further reports that t-SNE visualizations show similar clustering of relation patterns across tasks such as 5m_vs_6m and 8m_vs_9m, with additional isolated clusters appearing as entity scale increases. This is presented as evidence that relation patterns retain task-agnostic structure while also exhibiting richer diversity in more complex settings.

3. Task-conditioned decision-making via a conditional hypernetwork

RPG does not directly map relation patterns to fixed action heads. Instead, it uses a conditional hypernetwork to generate a task-specific decision-maker from a learnable task embedding $\mathcal{N}_c \subseteq \mathcal{M}_c$6. For task $\mathcal{N}_c \subseteq \mathcal{M}_c$7, the embedding is softly initialized as

$\mathcal{N}_c \subseteq \mathcal{M}_c$8

where $\mathcal{N}_c \subseteq \mathcal{M}_c$9 is Gaussian-initialized and $s_t \in \mathcal{S}_c$0 controls retention of previous task identity. For the first task, $s_t \in \mathcal{S}_c$1 is Gaussian-initialized and learnable (Yao et al., 8 Jul 2025).

The hypernetwork produces two parameter sets: $s_t \in \mathcal{S}_c$2 This split reflects the paper’s decomposition of the action space into ego actions, which belong to a fixed subspace independent of entity cardinalities, and interaction actions, whose dimensionality varies with the number of other entities.

Given the relational history $s_t \in \mathcal{S}_c$3, the decision-maker computes ego-action values as

$s_t \in \mathcal{S}_c$4

For interaction actions, it concatenates $s_t \in \mathcal{S}_c$5 with each other entity’s features $s_t \in \mathcal{S}_c$6 and produces entity-wise action-values: $s_t \in \mathcal{S}_c$7 The full local action-value vector is

$s_t \in \mathcal{S}_c$8

This design allows a fixed relational representation to be reused across tasks while varying the final decision layer according to task identity and action-space structure. The paper explicitly contrasts this with approaches that rely on static heads or progressive expansion. A plausible implication is that RPG treats task variability primarily as a change in how relational abstractions should be decoded, rather than as a need to relearn the abstractions themselves.

The local Q-values are integrated by a QMIX-style mixing network. RPG extends the mixing stage with multi-head attention to handle varying global-state dimensions across tasks, thereby maintaining centralized coordination under CTDE even when entity sets are dynamic. The exact mixing architecture is deferred to the appendix of the paper, but the stated role of the attention module is to accommodate variable state dimensionality while preserving value factorization.

4. Anti-forgetting mechanisms and training objective

RPG combines standard temporal-difference learning with regularizers on both the relation capturer and the hypernetwork. The base RL objective is

$s_t \in \mathcal{S}_c$9

To reduce forgetting in the relation capturer, the paper adopts a Taylor-based parameter-importance estimate inspired by Taylor pruning. The exact zeroing-based importance computation is described as prohibitive, so the method uses a second-order expansion: $o_t^i \in \mathcal{O}_c$0 In practice, only the quadratic term is retained, with importance averaged over $o_t^i \in \mathcal{O}_c$1 samples from the previous buffer and normalized: $o_t^i \in \mathcal{O}_c$2 Accumulated importance matrices from previous tasks are used in the capturer regularizer

$o_t^i \in \mathcal{O}_c$3

where $o_t^i \in \mathcal{O}_c$4 is the current parameter and $o_t^i \in \mathcal{O}_c$5 is the parameter after task $o_t^i \in \mathcal{O}_c$6.

This regularization is modulated by a return-aware discount computed from normalized early episodic return: $o_t^i \in \mathcal{O}_c$7 A larger $o_t^i \in \mathcal{O}_c$8 indicates good adaptation on the new task and therefore stronger retention of prior relation patterns; a smaller $o_t^i \in \mathcal{O}_c$9 relaxes the constraint to permit learning new relation patterns when the new observation space is more complex.

The hypernetwork is regularized separately to preserve previously generated task-specific decision-makers: $i$0 where $i$1 denotes hypernetwork parameters frozen after completing task $i$2. This explicitly constrains the outputs produced for past task embeddings while still allowing the hypernetwork to adapt on the current task.

The total loss for task $i$3 is

$i$4

Here $i$5, $i$6, $i$7, and $i$8 are hyperparameters controlling attention sparsity, capturer retention, hypernetwork retention, and task-embedding initialization.

The training protocol follows the sequence: initialize $i$9; collect decentralized episodes while computing $a_t^i \in \mathcal{A}_c$0; generate $a_t^i \in \mathcal{A}_c$1 and $a_t^i \in \mathcal{A}_c$2; compute local Q-values; mix them with the global state to obtain $a_t^i \in \mathcal{A}_c$3; compute $a_t^i \in \mathcal{A}_c$4 and $a_t^i \in \mathcal{A}_c$5; if $a_t^i \in \mathcal{A}_c$6, compute $a_t^i \in \mathcal{A}_c$7 and $a_t^i \in \mathcal{A}_c$8; optimize $a_t^i \in \mathcal{A}_c$9; then freeze a copy of the hypernetwork and store capturer parameters for future importance estimation. Continual evaluation is performed on all prior tasks after each new task is learned. Experiments average results over 3 random seeds (Yao et al., 8 Jul 2025).

5. Empirical performance on SMAC and LBF

RPG is evaluated on SMAC and LBF. In the continual SMAC sequence, the tasks are $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$0, where observation and action spaces vary with the number of entities. In LBF, agents operate in a 2D grid and collect food when the sum of agent levels is at least the food level; larger maps and more entities increase task complexity. Baselines for continual learning on QMIX backbones are MACPro, EWC, MAS, Replay, L2, and Finetuning. Additional single-task baselines are reported for SMAC and LBF, including VDN, QMIX, WQMIX, QTRAN, ResQ, HPN, FtQMIX, MAA2C, COMA, MADDPG, MAPPO, and IPPO (Yao et al., 8 Jul 2025).

The paper reports that RPG prevents catastrophic forgetting while adapting to new tasks more effectively than the baselines. In the training curves, EWC, MAS, and Replay degrade after the first task and fail to reach optimal performance on new tasks. Finetuning exhibits severe forgetting, while L2 overly constrains learning. MACPro is described as the strongest baseline, but still inferior to RPG in both single-task performance and plasticity. RPG also adapts faster to 8m_vs_9m and uses fewer parameters: approximately 112K agent parameters versus MACPro’s 377K.

Ablation results on plasticity after tasks $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$1 are reported as follows:

Variant	Plasticity values
RPG	0.875, 1.00, 1.00, 0.91
w/o $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$2	0.86, 1.00, 1.00, 0.95
w/o $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$3	0.84, 1.00, 1.00, 0.875
w/o $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$4	0.865, 1.00, 1.00, 0.86
RPG(EWC)	0.875, 1.00, 1.00, 0.93
RPG(MLP)	0.84, 1.00, 1.00, 0.59

The paper interprets these results as follows. Removing $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$5 increases plasticity but harms forgetting resistance, while replacing that term with EWC yields similar degradation and is used to validate the Taylor-based importance estimate. Removing $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$6 worsens forgetting, and replacing the hypernetwork decision-maker with an MLP sharply reduces plasticity on 8m_vs_9m to 0.59, supporting the claim that task-specific generation by the hypernetwork is crucial. Removing both regularizers weakens forgetting resistance.

In single-task settings, RPG achieves the best performance on several super-hard SMAC maps and outperforms baselines in LBF, especially when more entities and larger areas are present. A sparse-attention ablation shows that removing $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$7 slightly reduces performance, which the paper attributes to the benefit of focusing on a few key entities.

Zero-shot generalization is evaluated by training on the continual SMAC sequence and testing on unseen tasks $$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$8. The reported results are:

Unseen task	RPG	EWC	MAS
3m	$$\boldsymbol{a}_t = \{a_t^i\}_{i=1}^{|\mathcal{N}_c|}$$9	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$0	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$1
8m	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$2	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$3	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$4
10m	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$5	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$6	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$7
10m_vs_11m	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$8	$\mathcal{P}_c(s_{t+1}\mid s_t,\boldsymbol{a}_t)$9	$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$00
15m	$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$01	$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$02	$$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$03

RPG is best on 4 of the 5 unseen tasks and competitive on 3m. The paper notes that the 3m result suggests that, for very simple tasks, complex relational reasoning may be less advantageous than lighter regularization mechanisms.

6. Interpretation, related methods, and limitations

RPG is positioned as a Co-MARL method that explicitly separates relational abstraction from task-conditioned action realization. In this respect, it differs from QMIX, WQMIX, and QTRAN by focusing on task-agnostic relational extraction combined with task-conditioned decision generation. It also uses hypernetworks in a broader sense than QMIX’s mixing hypernets and MAVEN’s exploration mechanism, because RPG generates decision-layer parameters rather than only conditioning the mixing stage. Compared with Concord, the paper emphasizes support for dynamic entity cardinalities and explicit anti-forgetting regularization. Compared with MACPro, which uses a shared feature extractor with dynamic policy heads, RPG places the inductive bias on entity-wise relations through attention and preserves those relations with Taylor-based importance, while the hypernetwork supplies task-conditioned plasticity without progressive expansion (Yao et al., 8 Jul 2025).

The method’s analysis supports three main interpretations. First, relation patterns appear to be reusable across tasks, as indicated by similar t-SNE clustering across different SMAC scenarios. Second, similar relation patterns can still induce different decision preferences, reflected in different Q-value distributions after mapping into different action spaces. Third, the stability-plasticity dilemma is handled by distributing retention mechanisms across two levels: the capturer preserves relational structure, while the hypernetwork preserves task-specific decoders.

The reported complexity profile is comparatively compact. RPG uses approximately 112K agent parameters, substantially fewer than MACPro’s 377K, and avoids progressive policy-head growth across tasks. The overhead consists of storing the frozen hypernetwork $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$04, storing importance matrices $$\mathcal{Z}_c = \langle c, \mathcal{N}_c, \mathcal{M}_c, \mathcal{S}_c, \mathcal{O}_c, \mathcal{A}_c, \mathcal{R}_c, \mathcal{P}_c, \gamma \rangle,$$05 for capturer parameters across past tasks, and computing Taylor gradients on a small buffer.

The stated limitations are specific. RPG currently focuses on homogeneous cooperative teams and does not address perpetual learning or heterogeneous agent types. The zero-shot result on 3m indicates that the relational machinery may be unnecessary or even suboptimal in very simple settings. This suggests that RPG is best understood not as a universal replacement for simpler continual MARL methods, but as a method tailored to environments where dynamic entity structure and action-space variability make explicit relational abstraction advantageous.

Taken together, RPG advances continual multi-agent coordination by formalizing relation patterns as a transferable substrate and by linking them to task-specific decision-making through conditional hypernetwork generation. Its central claim is that continual coordination benefits from preserving relational structure separately from preserving action-space-specific decoders, and the empirical results on SMAC and LBF support that decomposition as an effective response to catastrophic forgetting.