MA-SPL: Multi-Agent Surrogate Policy Learning

• The paper introduces MA-SPL, a distributed algorithm that uses a policy-based continuous extension to achieve a (1-c/e)-approximation guarantee for monotone submodular objectives.
• MA-SPL extends online coordination to monotone α-weakly DR-submodular and (γ,β)-weakly submodular objectives through surrogate gradient ascent and a novel lossless rounding scheme.
• The decentralized structure integrates local marginal oracles, consensus-based policy updates, and stochastic surrogate gradients to efficiently solve multi-agent coordination with limited feedback.

MA-SPL denotes Multi-Agent Surrogate Policy Learning, the first of two online algorithms introduced for the multi-agent online coordination (MA-OC) problem. In that formulation, multiple agents repeatedly choose one action each from disjoint private action sets, while the environment reveals a time-varying set utility after the joint action is taken. MA-SPL is designed both to attain the optimal $\left(1-\frac{c}{e}\right)$-approximation guarantee for monotone submodular objectives with curvature $c$ and to extend online multi-agent coordination to monotone $\alpha$-weakly DR-submodular and monotone $(\gamma,\beta)$-weakly submodular objectives through a new relaxation called the policy-based continuous extension (Zhang et al., 26 Sep 2025).

1. Problem setting and formal objective

In MA-OC, agents are indexed by

$\mathcal{N}\triangleq \{1,\dots,n\},$

and communicate over an undirected graph $G(\mathcal{N},\mathcal{E})$. Agent $i$ has a private action set

$$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$

with disjointness

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$

and global ground set

$$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$

At each round $c$0, every agent $c$1 selects one action $c$2. The environment then reveals a utility set function

$c$3

and the team receives reward

$c$4

The per-round combinatorial problem is

$c$5

The feedback model is local. After $c$6 is revealed, agent $c$7 may query only a local marginal oracle

$c$8

for $c$9. This restriction is central: MA-SPL is not formulated as a centralized full-information method.

Performance is measured through dynamic $\alpha$0-regret,

$\alpha$1

where $\alpha$2 is the exact optimizer of (1) at round $\alpha$3. The comparator variation is

$\alpha$4

This definition makes explicit that MA-SPL is evaluated against a time-varying benchmark sequence rather than a single static solution (Zhang et al., 26 Sep 2025).

2. Policy representation and the policy-based continuous extension

MA-SPL assigns each agent a policy

$\alpha$5

where

$\alpha$6

The policy induces the sampling law

$\alpha$7

Thus each agent probabilistically chooses either one local action or $\alpha$8.

The paper’s central relaxation is the policy-based continuous extension

$\alpha$9

with feasible domain

$(\gamma,\beta)$0

Its defining advantage is that sampling from $(\gamma,\beta)$1 already respects the partition constraint $(\gamma,\beta)$2. The paper therefore describes it as providing a lossless rounding scheme for any set function, in contrast to multilinear-extension-based methods whose lossless rounding machinery is tied to submodularity. A direct consequence is

$(\gamma,\beta)$3

The extension also has an explicit gradient formula: $(\gamma,\beta)$4 For monotone $(\gamma,\beta)$5, these partial derivatives are nonnegative. This suggests that the continuous model preserves the marginal-utility structure of the discrete problem rather than merely approximating it abstractly (Zhang et al., 26 Sep 2025).

3. Objective classes and surrogate construction

The paper treats three nested objective classes. A normalized monotone set function satisfies

$(\gamma,\beta)$6

A submodular $(\gamma,\beta)$7 obeys

$(\gamma,\beta)$8

Its curvature is

$(\gamma,\beta)$9

A function is $\mathcal{N}\triangleq \{1,\dots,n\},$0-weakly DR-submodular if

$\mathcal{N}\triangleq \{1,\dots,n\},$1

and $\mathcal{N}\triangleq \{1,\dots,n\},$2-weakly submodular if it is simultaneously $\mathcal{N}\triangleq \{1,\dots,n\},$3-weakly submodular from below,

$\mathcal{N}\triangleq \{1,\dots,n\},$4

and $\mathcal{N}\triangleq \{1,\dots,n\},$5-weakly submodular from above,

$\mathcal{N}\triangleq \{1,\dots,n\},$6

The policy-based extension satisfies gradient-gap inequalities that connect the continuous gradient to discrete optimality. For monotone $\mathcal{N}\triangleq \{1,\dots,n\},$7-weakly DR-submodular $\mathcal{N}\triangleq \{1,\dots,n\},$8,

$\mathcal{N}\triangleq \{1,\dots,n\},$9

and for monotone $G(\mathcal{N},\mathcal{E})$0-weakly submodular $G(\mathcal{N},\mathcal{E})$1,

$G(\mathcal{N},\mathcal{E})$2

The paper then shows that optimizing $G(\mathcal{N},\mathcal{E})$3 directly yields suboptimal stationary-point factors. MA-SPL therefore maximizes a surrogate $G(\mathcal{N},\mathcal{E})$4 whose gradient is

$G(\mathcal{N},\mathcal{E})$5

with a class-dependent weight $G(\mathcal{N},\mathcal{E})$6. The choices are: $G(\mathcal{N},\mathcal{E})$7 for monotone $G(\mathcal{N},\mathcal{E})$8-weakly DR-submodular objectives;

$G(\mathcal{N},\mathcal{E})$9

for monotone $i$0-weakly submodular objectives; and

$i$1

for submodular objectives with curvature $i$2, together with an extra linear term

$i$3

This surrogate design is what gives MA-SPL its approximation guarantees. At surrogate stationary points, the paper derives

$i$4

for monotone submodular functions with curvature $i$5,

$i$6

for monotone $i$7-weakly DR-submodular functions, and

$i$8

for monotone $i$9-weakly submodular functions. The first factor is presented as the optimal $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$0-approximation guarantee (Zhang et al., 26 Sep 2025).

4. Distributed algorithmic structure

MA-SPL is decentralized. Each agent $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$1 maintains local estimates

$$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$2

of all agents’ policies. Initialization is

$$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$3

At round $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$4, agent $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$5 first normalizes its own policy,

$$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$6

samples $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$7, and executes it. Agents then exchange their local policy vectors with neighbors. This neighbor exchange is the mechanism by which a distributed estimate of the global policy profile is maintained.

To estimate the surrogate gradient, agent $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$8 samples $$\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},$$9 from a random variable $$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$0 with cumulative distribution

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$1

It then samples surrogate actions $$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$2 using $$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$3, forms

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$4

and computes the coordinate-wise stochastic estimator

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$5

In the submodular-with-curvature case, this is modified by

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$6

The policy update couples consensus and projected ascent. For other agents’ estimates,

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$7

For its own policy,

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$8

followed by projection

$$\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,$$9

This combination of stochastic surrogate gradients, consensus mixing, and Euclidean projection is the operational core of MA-SPL. A plausible implication is that the method should be read as a distributed projected stochastic-ascent scheme specialized to the policy-based extension rather than as a combinatorial online greedy rule (Zhang et al., 26 Sep 2025).

5. Guarantees, complexity, and relation to MA-MPL

The network assumptions are standard but explicit. The communication graph must be connected, and the mixing matrix $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$0 must be symmetric and doubly stochastic: $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$1 Let

$$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$2

If

$$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$3

then MA-SPL achieves

$$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$4

with

$$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$5

for the submodular, $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$6-weakly DR-submodular, and $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$7-weakly submodular cases, respectively (Zhang et al., 26 Sep 2025).

The paper also introduces MA-MPL as a parameter-free alternative. The motivation is explicit: MA-SPL requires $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$8, or $$\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.$$9, to choose the surrogate weight $c$00 in the weakly submodular settings, and those parameters are generally hard to compute exactly. MA-MPL removes that dependence at the cost of higher communication and oracle complexity.

Algorithm	Parameter dependence	Online costs and regret
MA-SPL	Needs $c$01 or $c$02 in weakly submodular cases	$c$03 communications, $c$04 queries, $c$05 regret
MA-MPL	Parameter-free	$c$06 communications, $c$07 queries, $c$08 regret

For monotone $c$09-weakly DR-submodular and monotone $c$10-weakly submodular objectives, MA-MPL is stated to maintain the same approximation ratio as MA-SPL. The trade-off is therefore not one of approximation quality but of parameter dependence versus online cost. This suggests that MA-SPL is the lower-complexity choice when curvature- or weak-submodularity information is available, whereas MA-MPL is the robust fallback when it is not (Zhang et al., 26 Sep 2025).

6. Empirical behavior, limitations, and acronym ambiguity

The experimental section studies two tracking-style coordination problems. In the first, a facility-location objective is used in a setting with 20 agents, 30 moving targets, and $c$11 iterations: $c$12 This objective is monotone submodular. The reported result is that MA-SPL significantly outperforms MA-OSMA and MA-OSEA on average utility, while also reducing the average distance to the nearest targets and increasing the number of targets within distance $c$13 (Zhang et al., 26 Sep 2025).

In the second experiment, the objective is

$c$14

which the paper treats as monotone $c$15-weakly DR-submodular and $c$16-weakly submodular. Here, both MA-SPL and MA-MPL substantially outperform RANDOM. At the same time, the paper stresses a limitation of MA-SPL: its performance depends on the chosen $c$17, and MA-MPL can outperform MA-SPL when MA-SPL uses only a coarse search over $c$18 values. Additional stated limitations are that MA-SPL is not projection-free, requires a connected graph with a symmetric doubly stochastic mixing matrix, and assumes local marginal-oracle access after each round (Zhang et al., 26 Sep 2025).

The acronym itself is potentially ambiguous in the broader arXiv literature. A common misconception is to identify MA-SPL with nearby but distinct terms. The paper introducing the Mass Agreement Score (MAS) explicitly states that it does not define a quantity called MA-SPL (Wiredu-Aidoo, 24 Mar 2026). The QSAR paper on SPL-Logsum likewise states that MA-SPL is not mentioned anywhere in the paper, and that its self-paced component is a standard hard-threshold binary SPL mechanism (Xia et al., 2018). The paper on Socratic Playground for Learning (SPL) also states that it does not explicitly mention a system, model, or variant called MA-SPL, though it speculates about future multiple roles and agents (Zhang et al., 2024). Within the material considered here, the explicit technical meaning of MA-SPL is therefore the one given in multi-agent online coordination: Multi-Agent Surrogate Policy Learning.