---
title: 'MA-SPL: Multi-Agent Surrogate Policy Learning'
url: https://www.emergentmind.com/topics/ma-spl
type: topic
---

# MA-SPL: Multi-Agent Surrogate Policy Learning

MA-SPL denotes **Multi-Agent Surrogate Policy Learning**, the first of two online algorithms introduced for the **multi-agent online coordination (MA-OC)** problem. In that formulation, multiple agents repeatedly choose one action each from disjoint private action sets, while the environment reveals a time-varying set utility after the joint action is taken. MA-SPL is designed both to attain the optimal \(\left(1-\frac{c}{e}\right)\)-approximation guarantee for monotone submodular objectives with curvature \(c\) and to extend online multi-agent coordination to monotone \(\alpha\)-weakly DR-submodular and monotone \((\gamma,\beta)\)-weakly submodular objectives through a new relaxation called the **policy-based continuous extension** [2509.22596].

## 1. Problem setting and formal objective

In MA-OC, agents are indexed by
\[
\mathcal{N}\triangleq \{1,\dots,n\},
\]
and communicate over an undirected graph \(G(\mathcal{N},\mathcal{E})\). Agent \(i\) has a private action set
\[
\mathcal{V}_i \triangleq \{v_{i,1},\dots,v_{i,\kappa_i}\},
\]
with disjointness
\[
\mathcal{V}_i \cap \mathcal{V}_j = \emptyset,\quad i\neq j,
\]
and global ground set
\[
\mathcal{V}\triangleq \bigcup_{i\in\mathcal{N}} \mathcal{V}_i.
\]

At each round \(t\in[T]\), every agent \(i\) selects one action \(a_i(t)\in \mathcal{V}_i\). The environment then reveals a utility set function
\[
f_t:2^{\mathcal V}\to \mathbb{R}_+,
\]
and the team receives reward
\[
f_t\!\left(\bigcup_{i\in\mathcal N}\{a_i(t)\}\right).
\]
The per-round combinatorial problem is
\[
\max f_t(S),\qquad \text{s.t. } S\subseteq \mathcal V,\ \ |S\cap \mathcal V_i|\le 1,\ \forall i\in\mathcal N.
\tag{1}
\]

The feedback model is local. After \(f_t\) is revealed, agent \(i\) may query only a local marginal oracle
\[
\mathcal{Q}_t^i:\mathcal V_i\times 2^{\mathcal V}\to \mathbb R_+, \qquad 
\mathcal Q_t^i(a,S)\triangleq f_t(a\mid S)\triangleq f_t(S\cup\{a\})-f_t(S),
\]
for \(a\in\mathcal V_i\). This restriction is central: MA-SPL is not formulated as a centralized full-information method.

Performance is measured through dynamic \(\rho\)-regret,
\[
R_\rho^*(T)\triangleq \rho\sum_{t=1}^T f_t(\mathcal A_t^*)-
\sum_{t=1}^T f_t\!\left(\bigcup_{i\in\mathcal N}\{a_i(t)\}\right),
\]
where \(\mathcal A_t^*\) is the exact optimizer of (1) at round \(t\). The comparator variation is
\[
P_T \triangleq \sum_{t=2}^T |\mathcal A_t^* \triangle \mathcal A_{t-1}^*|.
\]
This definition makes explicit that MA-SPL is evaluated against a time-varying benchmark sequence rather than a single static solution [2509.22596].

## 2. Policy representation and the policy-based continuous extension

MA-SPL assigns each agent a policy
\[
\boldsymbol{\pi}_i = (\pi_{i,1},\dots,\pi_{i,\kappa_i}) \in \Delta_{\kappa_i},
\]
where
\[
\Delta_m \triangleq \{(x_1,\dots,x_m)\mid \sum_{i=1}^m x_i \le 1,\ x_i\ge 0\}.
\]
The policy induces the sampling law
\[
p(v_{i,m}\mid \boldsymbol\pi_i)=\pi_{i,m},\qquad
p(\emptyset\mid \boldsymbol\pi_i)=1-\sum_{m=1}^{\kappa_i}\pi_{i,m}.
\]
Thus each agent probabilistically chooses either one local action or \(\emptyset\).

The paper’s central relaxation is the **policy-based continuous extension**
\[
F_t(\boldsymbol\pi_1,\dots,\boldsymbol\pi_n) \triangleq 
\sum_{a_i\in \mathcal V_i\cup\{\emptyset\},\forall i\in\mathcal N}
\left(
f_t\!\left(\bigcup_{i=1}^n\{a_i\}\right)
\prod_{i=1}^n p(a_i\mid \boldsymbol\pi_i)
\right),
\tag{2}
\]
with feasible domain
\[
\max F_t(\boldsymbol\pi_1,\dots,\boldsymbol\pi_n), \qquad 
\text{s.t. }\|\boldsymbol\pi_i\|_1\le 1,\ \boldsymbol\pi_i\in[0,1]^{\kappa_i},\ \forall i\in\mathcal N.
\tag{3}
\]

Its defining advantage is that sampling from \((\boldsymbol\pi_1,\dots,\boldsymbol\pi_n)\) already respects the partition constraint \(|S\cap \mathcal V_i|\le 1\). The paper therefore describes it as providing a **lossless rounding scheme for any set function**, in contrast to multilinear-extension-based methods whose lossless rounding machinery is tied to submodularity. A direct consequence is
\[
\mathbb E\!\left[f_t\!\left(\bigcup_i\{a_i\}\right)\right]
=
F_t(\boldsymbol\pi_1,\dots,\boldsymbol\pi_n).
\]

The extension also has an explicit gradient formula:
\[
\frac{\partial F_t}{\partial \pi_{i,m}}(\boldsymbol\pi_1,\dots,\boldsymbol\pi_n)
=
\mathbb E_{a_j\sim \boldsymbol\pi_j,\forall j\in\mathcal N}
\Big(
f_t\big(v_{i,m}\mid \cup_{j\neq i}\{a_j\}\big)
\Big).
\]
For monotone \(f_t\), these partial derivatives are nonnegative. This suggests that the continuous model preserves the marginal-utility structure of the discrete problem rather than merely approximating it abstractly [2509.22596].

## 3. Objective classes and surrogate construction

The paper treats three nested objective classes. A normalized monotone set function satisfies
\[
f(\emptyset)=0,\qquad f(S)\le f(T)\ \text{for}\ S\subseteq T.
\]
A submodular \(f\) obeys
\[
f(v\mid S)\ge f(v\mid T), \qquad \forall S\subseteq T\subseteq \mathcal V,\ \forall v\in\mathcal V\setminus T.
\]
Its curvature is
\[
c \triangleq 1-\min_{S\subseteq \mathcal V,\ v\notin S}
\frac{f(S\cup\{v\})-f(S)}{f(\{v\})-f(\emptyset)}.
\]

A function is \(\alpha\)-weakly DR-submodular if
\[
f(v\mid S)\ge \alpha f(v\mid T),
\qquad \forall S\subseteq T\subseteq \mathcal V,\ \forall v\in\mathcal V\setminus T,
\]
and \((\gamma,\beta)\)-weakly submodular if it is simultaneously \(\gamma\)-weakly submodular from below,
\[
\sum_{v\in T\setminus S} f(v\mid S)\ge \gamma\big(f(T)-f(S)\big),
\qquad \forall S\subseteq T\subseteq \mathcal V,
\]
and \(\beta\)-weakly submodular from above,
\[
\sum_{v\in T\setminus S} f(v\mid T\setminus\{v\})
\le \beta\big(f(T)-f(S)\big),
\qquad \forall S\subseteq T\subseteq \mathcal V.
\]

The policy-based extension satisfies gradient-gap inequalities that connect the continuous gradient to discrete optimality. For monotone \(\alpha\)-weakly DR-submodular \(f_t\),
\[
\alpha\Big(f_t(S)-F_t(\boldsymbol\pi)\Big)
\le
\sum_{(i,m):v_{i,m}\in S}
\frac{\partial F_t}{\partial \pi_{i,m}}(\boldsymbol\pi),
\tag{4}
\]
and for monotone \((\gamma,\beta)\)-weakly submodular \(f_t\),
\[
\gamma^2 f_t(S)-(\beta(1-\gamma)+\gamma^2)F_t(\boldsymbol\pi)
\le
\sum_{(i,m):v_{i,m}\in S}
\frac{\partial F_t}{\partial \pi_{i,m}}(\boldsymbol\pi).
\tag{5}
\]

The paper then shows that optimizing \(F_t\) directly yields suboptimal stationary-point factors. MA-SPL therefore maximizes a surrogate \(F_t^s\) whose gradient is
\[
\nabla F_t^s(\mathbf x)=\int_0^1 w(z)\nabla F_t(z*\mathbf x)\,dz,
\]
with a class-dependent weight \(w(z)\). The choices are:
\[
w(z)=e^{\alpha(z-1)}
\]
for monotone \(\alpha\)-weakly DR-submodular objectives;
\[
w(z)=e^{\phi(\gamma,\beta)(z-1)},\qquad
\phi(\gamma,\beta)=\beta(1-\gamma)+\gamma^2
\]
for monotone \((\gamma,\beta)\)-weakly submodular objectives; and
\[
w(z)=e^{z-1}
\]
for submodular objectives with curvature \(c\), together with an extra linear term
\[
G_t(\boldsymbol\pi_1,\dots,\boldsymbol\pi_n)
\triangleq
\sum_{i=1}^n\sum_{m=1}^{\kappa_i}
\big(f_t(v_{i,m}\mid \mathcal V-\{v_{i,m}\})\big)\pi_{i,m}.
\]

This surrogate design is what gives MA-SPL its approximation guarantees. At surrogate stationary points, the paper derives
\[
F_t(\pi^s)\ge \left(1-\frac{c}{e}\right)f_t(S^*)
\]
for monotone submodular functions with curvature \(c\),
\[
F_t(\pi^s)\ge (1-e^{-\alpha})f_t(S^*)
\]
for monotone \(\alpha\)-weakly DR-submodular functions, and
\[
F_t(\pi^s)\ge 
\frac{\gamma^2(1-e^{-\phi(\gamma,\beta)})}{\phi(\gamma,\beta)}f_t(S^*)
\]
for monotone \((\gamma,\beta)\)-weakly submodular functions. The first factor is presented as the **optimal** \(\left(1-\frac{c}{e}\right)\)-approximation guarantee [2509.22596].

## 4. Distributed algorithmic structure

MA-SPL is decentralized. Each agent \(i\) maintains local estimates
\[
(\boldsymbol\pi_{i,1}(t),\dots,\boldsymbol\pi_{i,n}(t))
\]
of all agents’ policies. Initialization is
\[
\boldsymbol\pi_{i,i}(1)=\frac{1}{\kappa_i}\mathbf 1_{\kappa_i},\qquad
\boldsymbol\pi_{i,j}(1)=\mathbf 0_{\kappa_j}\ (j\neq i).
\]

At round \(t\), agent \(i\) first normalizes its own policy,
\[
\mathbf p_i(t)\triangleq \frac{\boldsymbol\pi_{i,i}(t)}{\|\boldsymbol\pi_{i,i}(t)\|_1},
\]
samples \(a_i(t)\in\mathcal V_i\), and executes it. Agents then exchange their local policy vectors with neighbors. This neighbor exchange is the mechanism by which a distributed estimate of the global policy profile is maintained.

To estimate the surrogate gradient, agent \(i\) samples \(z_i(t)\) from a random variable \(\mathcal Z\) with cumulative distribution
\[
\Pr(\mathcal Z\le z)=
\frac{\int_0^z w(a)\,da}{\int_0^1 w(a)\,da},
\qquad z\in[0,1].
\]
It then samples surrogate actions \(\widetilde a_j(t)\in\mathcal V_j\cup\{\emptyset\}\) using \(z_i(t)*\boldsymbol\pi_{i,j}\), forms
\[
S_i(t)\triangleq \bigcup_{j\neq i}\{\widetilde a_j(t)\},
\]
and computes the coordinate-wise stochastic estimator
\[
d_{i,m}(t)
=
\left(\int_0^1 w(z)\,dz\right) f_t(v_{i,m}\mid S_i(t)),
\qquad \forall m\in[\kappa_i].
\]
In the submodular-with-curvature case, this is modified by
\[
d_{i,m}(t) \leftarrow d_{i,m}(t)+e^{-1}f_t(v_{i,m}\mid \mathcal V-\{v_{i,m}\}).
\]

The policy update couples consensus and projected ascent. For other agents’ estimates,
\[
\boldsymbol\pi_{i,j}(t+1)
\triangleq
\sum_{k\in\mathcal N_i\cup\{i\}} w_{ik}\boldsymbol\pi_{k,j}(t),
\qquad j\neq i.
\]
For its own policy,
\[
\mathbf y_{i,i}(t+1)
\triangleq
\sum_{j\in\mathcal N_i\cup\{i\}} w_{ij}\boldsymbol\pi_{j,i}(t) +\eta_t \mathbf d_i(t),
\]
followed by projection
\[
\boldsymbol\pi_{i,i}(t+1)
\triangleq
\arg\min_{\mathbf b\in \Delta_{\kappa_i}}
\|\mathbf b-\mathbf y_{i,i}(t+1)\|_2.
\]

This combination of stochastic surrogate gradients, consensus mixing, and Euclidean projection is the operational core of MA-SPL. A plausible implication is that the method should be read as a distributed projected stochastic-ascent scheme specialized to the policy-based extension rather than as a combinatorial online greedy rule [2509.22596].

## 5. Guarantees, complexity, and relation to MA-MPL

The network assumptions are standard but explicit. The communication graph must be connected, and the mixing matrix \(\mathbf W\) must be symmetric and doubly stochastic:
\[
\mathbf W^\top=\mathbf W,\qquad \mathbf W\mathbf 1_n=\mathbf 1_n.
\]
Let
\[
\tau=\max(|\lambda_2(\mathbf W)|,|\lambda_n(\mathbf W)|)<1.
\]
If
\[
\eta_t=\mathcal O\!\left(\sqrt{\frac{(1-\tau)P_T}{T}}\right),
\]
then MA-SPL achieves
\[
\mathbb E[R_\rho^*(T)]
\le
\mathcal O\!\left(\sqrt{\frac{P_T T}{1-\tau}}\right),
\]
with
\[
\rho=1-\frac{c}{e},
\qquad
\rho=1-e^{-\alpha},
\qquad
\rho=
\frac{\gamma^2(1-e^{-(\beta(1-\gamma)+\gamma^2)})}{\beta(1-\gamma)+\gamma^2},
\]
for the submodular, \(\alpha\)-weakly DR-submodular, and \((\gamma,\beta)\)-weakly submodular cases, respectively [2509.22596].

The paper also introduces **MA-MPL** as a parameter-free alternative. The motivation is explicit: MA-SPL requires \(\alpha\), or \((\gamma,\beta)\), to choose the surrogate weight \(w(z)\) in the weakly submodular settings, and those parameters are generally hard to compute exactly. MA-MPL removes that dependence at the cost of higher communication and oracle complexity.

| Algorithm | Parameter dependence | Online costs and regret |
|---|---|---|
| **MA-SPL** | Needs \(\alpha\) or \((\gamma,\beta)\) in weakly submodular cases | \(\mathcal O(T)\) communications, \(\mathcal O(\kappa T)\) queries, \(\mathcal O\!\left(\sqrt{\frac{P_TT}{1-\tau}}\right)\) regret |
| **MA-MPL** | Parameter-free | \(\mathcal O(T^{3/2})\) communications, \(\mathcal O(\kappa T^{5/2})\) queries, \(\mathcal O(d(G)\sqrt{P_TT})\) regret |

For monotone \(\alpha\)-weakly DR-submodular and monotone \((\gamma,\beta)\)-weakly submodular objectives, MA-MPL is stated to maintain the same approximation ratio as MA-SPL. The trade-off is therefore not one of approximation quality but of parameter dependence versus online cost. This suggests that MA-SPL is the lower-complexity choice when curvature- or weak-submodularity information is available, whereas MA-MPL is the robust fallback when it is not [2509.22596].

## 6. Empirical behavior, limitations, and acronym ambiguity

The experimental section studies two tracking-style coordination problems. In the first, a facility-location objective is used in a setting with **20 agents**, **30 moving targets**, and **\(T=1250\)** iterations:
\[
f_t(S)=\sum_{j=1}^{30}\max_{(\theta,s,i)\in S}\frac{1}{\|o_{(\theta,s,i)}(t)-o_j(t)\|_2}.
\]
This objective is monotone submodular. The reported result is that **MA-SPL significantly outperforms MA-OSMA and MA-OSEA** on average utility, while also reducing the average distance to the nearest targets and increasing the number of targets within distance \(1\) [2509.22596].

In the second experiment, the objective is
\[
f_t(S)=\sum_{j=1}^{30}\left(\operatorname{Tr}(I_j^{-1}(t))-\operatorname{Tr}(B_S^j(t))\right),
\]
which the paper treats as monotone \(\alpha\)-weakly DR-submodular and \((\gamma,\beta)\)-weakly submodular. Here, **both MA-SPL and MA-MPL substantially outperform RANDOM**. At the same time, the paper stresses a limitation of MA-SPL: its performance depends on the chosen \(\alpha\), and MA-MPL can outperform MA-SPL when MA-SPL uses only a coarse search over \(\alpha\) values. Additional stated limitations are that MA-SPL is not projection-free, requires a connected graph with a symmetric doubly stochastic mixing matrix, and assumes local marginal-oracle access after each round [2509.22596].

The acronym itself is potentially ambiguous in the broader arXiv literature. A common misconception is to identify MA-SPL with nearby but distinct terms. The paper introducing the **Mass Agreement Score (MAS)** explicitly states that it does **not** define a quantity called MA-SPL [2603.23581]. The QSAR paper on **SPL-Logsum** likewise states that **MA-SPL is not mentioned anywhere in the paper**, and that its self-paced component is a standard hard-threshold binary SPL mechanism [1804.08615]. The paper on **Socratic Playground for Learning (SPL)** also states that it does **not** explicitly mention a system, model, or variant called MA-SPL, though it speculates about future multiple roles and agents [2406.13919]. Within the material considered here, the explicit technical meaning of **MA-SPL** is therefore the one given in multi-agent online coordination: **Multi-Agent Surrogate Policy Learning**.

Source: https://www.emergentmind.com/topics/ma-spl