Submodular Policy Learning for Distributed Task Allocation in Open Multi-Agent Systems

Abstract: This paper studies policy learning for distributed task allocation in open multi-agent systems, where agents may join and leave in a time-varying fashion, with submodular stage team utilities. At each time, the active agents select actions from local categorical policies such that the feasible joint agent-action pairs form a partition matroid. Standard continuous relaxations of submodular set functions are based on independent Bernoulli sampling, making them inconsistent with agents' policies.To solve this mismatch, we propose the \emph{partition multilinear extension} (PME), a policy-based relaxation whose continuous support matches feasible actions under categorical policies.We prove that the marginal gains of the stage utility provide an unbiased estimator of the gradient of the PME and that maximizing the PME over action distributions is equivalent to maximizing the stage utilities over agent actions, which are critical to devise principled policy gradient.Building on this, we design \emph{SubMAPL}, a centralized-training decentralized-execution KL-mirror policy-learning method that uses local marginal gains as stochastic PME gradients during training. KL-mirror updates preserve categorical feasibility without Euclidean projection.In the case where agents run tabular-softmax policies, we introduce open policy migration and an open-system KL tracking variation to handle agent arrivals and departures. Using dynamic regret analysis, we establish a lower bound on the cumulative utility which accounts for the openness of the environment and for the gap between optimal stage-wise and global utilities. Simulations on multi-agent coverage demonstrate that SubMAPL outperforms policy-gradient and online-learning baselines.

• The paper introduces the partition multilinear extension and SubMAPL, a KL-mirror policy-learning algorithm that provides unbiased marginal-gain gradients for factorized categorical policies in open multi-agent systems.
• Theoretical analysis establishes a 1/2-approximation dynamic regret bound and, under time-expanded submodularity, a 1/3-approximation finite-horizon guarantee with an O(√T) approximation gap plus tracking variation.
• Experiments on decentralized grid coverage show SubMAPL achieving 0.963 normalized area under the coverage curve versus 0.924 for the strongest baseline, while reaching 95% coverage in 580 steps versus 904.

This paper develops a policy-learning framework for distributed task allocation in open multi-agent systems (OMAS), where the set of active agents varies over time and team utilities exhibit diminishing-returns (submodular) structure. The authors, Liu et al., introduce a policy-based continuous relaxation—the partition multilinear extension (PME)—and build a KL-mirror policy-learning algorithm, SubMAPL, with approximation-regret guarantees. The work addresses a genuine mismatch in prior literature: standard continuous relaxations of submodular set functions rely on independent Bernoulli sampling, which is inconsistent with the factorized categorical policies executed by decentralized agents (2608.14390).

Problem formulation

The paper models the setting as a finite-horizon, partially observed open multi-agent MDP with time-varying active-agent set $\mathcal N_t$, partitioned into remaining, arriving, and departing agents. At each step, the ground set $\Omega_t$ consists of agent-action pairs, and feasible joint actions form a partition matroid: each active agent selects exactly one action from its local set. Each agent acts via a factorized categorical policy conditioned on a local observation. The team utility $F_t(\cdot;s_t)$ is assumed normalized, monotone, and submodular over $\Omega_t$, capturing redundancy such as overlapping sensor footprints.

The objective is to learn decentralized categorical policies maximizing expected cumulative submodular utility over the horizon. Even a single time step reduces to monotone submodular maximization under a partition matroid, which is NP-hard, so exact optimization is out of reach and the analysis targets approximation guarantees. The authors adopt a stagewise approach, justified formally by an exact equivalence between the finite-horizon policy objective and the cumulative PME along the induced trajectory (Corollary 3.1 in the paper's numbering).

The partition multilinear extension

The PME replaces the Bernoulli sampling of the classical multilinear extension with partition sampling: under a marginal vector $x$ in the partition-matroid polytope, each agent independently selects an action with probability proportional to its coordinates, or selects no element with the residual probability. On the categorical-policy face $\mathcal F_t$, where each agent's coordinates sum to one, this sampling distribution coincides exactly with the factorized categorical execution model. The paper proves three key structural results:

• Objective equivalence: the expected stage utility under a factorized categorical policy equals the PME evaluated at the policy-induced marginal vector.
• Gradient characterization: the PME coordinate derivative with respect to $x_{(i,a)}$ equals the expected marginal gain $F_t((i,a)\mid A^{-i})$ under the sampling distribution of the other agents. This identity is the foundation for the credit-assignment scheme.
• Exactness on the categorical face: by monotonicity and multilinearity, the PME optimum over the polytope equals the discrete optimum over the matroid, and is attained on $\mathcal F_t$. Hence restricting to factorized categorical policies loses no stagewise optimal value.

The PME is further shown to be smooth, monotone, and DR-submodular on the polytope, inheriting the machinery of continuous submodular maximization. Compared to the policy-based extensions in prior online-coordination work, the PME formulation additionally handles Markovian dynamics and time-varying agent domains.

The SubMAPL algorithm

SubMAPL follows centralized training with decentralized execution (CTDE). Each agent maintains a tabular softmax policy over observation-action logits. After a joint action is sampled, for each active agent $i$ the method evaluates the marginal gain of every feasible action of $\Omega_t$0 against the same sampled actions of the other agents. This vector $\Omega_t$1 is an unbiased estimator of the PME gradient at the current policy—a stronger credit-assignment signal than sampled-action difference rewards, which evaluate only a single action per agent.

Policy updates use KL-regularized mirror ascent on the local simplex. A notable technical point is the exact equivalence between the KL-mirror update and an additive logit update under the tabular softmax parameterization: the multiplicative exponential-weights update on probabilities corresponds precisely to adding $\Omega_t$2 to the logits, so no Euclidean projection is needed and simplex feasibility is preserved by construction. The authors are careful to note that this exactness holds for the KL-mirror update; a standard softmax policy-gradient implementation would agree with PME ascent only to first order, which motivates their design choice.

To handle openness, the paper introduces open policy migration: remaining agents inherit their updated policies, arriving agents are initialized with finite logits (ensuring full support), and departing agents are dropped from the policy domain. This allows closed-system-trained policies to be reused whenever an agent reactivates, avoiding restarts after participation changes.

Theoretical guarantees

The analysis proceeds via dynamic approximation regret against time-varying stagewise optima. A new quantity, the open-system KL tracking variation $\Omega_t$3, measures increases in the KL potential between the stagewise optimal policy and the online policy, accounting for both comparator movement and changes in the policy domain induced by arrivals and departures. Under bounded marginal gains, the main regret result states

$\Omega_t$4

which balances a tracking term (small when the environment and comparator change slowly) against a $\Omega_t$5 drift term. With the standard step-size choice $\Omega_t$6, the regret scales as $\Omega_t$7 plus the tracking variation.

Connecting this stagewise bound to the finite-horizon objective $\Omega_t$8 requires care: the cumulative stagewise optima along the algorithm's own trajectory need not match the optimal policy value. In the general case, the paper obtains the decomposition $\Omega_t$9, where $F_t(\cdot;s_t)$0 is a benchmark-mismatch term that the stagewise analysis does not control. The authors state this plainly: the result is a decomposition, not a uniform approximation guarantee, unless additional temporal structure is available.

That structure is supplied by the time-expanded submodularity assumption, under which each stage utility is the marginal gain of a single policy-independent submodular function $F_t(\cdot;s_t)$1 over a time-expanded ground set—satisfied, for instance, by coverage of static maps with weighted targets. In that case the cumulative utility telescopes, and the paper proves the stronger bound $F_t(\cdot;s_t)$2. This is the paper's headline guarantee: a $F_t(\cdot;s_t)$3-approximation of the finite-horizon optimum, with the approximation gap shrinking at the $F_t(\cdot;s_t)$4 rate of the regret term. The proof combines a submodularity-based comparator inequality with the $F_t(\cdot;s_t)$5-factor stagewise regret, and the factor is tight for this style of argument. The authors note this improves their preliminary work, which gave only stagewise PME bounds without formal guarantees on the finite-horizon utility.

Numerical experiments

Experiments use a $F_t(\cdot;s_t)$6 grid coverage task with up to five agents, three information-density landscapes (uniform, Gaussian mixture, log-Gaussian process), and local observations restricted to a $F_t(\cdot;s_t)$7 region, at most two nearest neighbors, and local coverage status—so execution is genuinely decentralized with no consensus steps. Policies are trained in a closed system (3000 episodes, horizon 100) and evaluated on open profiles of length 2000.

Baselines are MA-SPL (policy-based continuous extension with projection), MA-OSEA (multilinear extension with curvature-aware gradients and KL updates), REINFORCE, and the centralized online greedy OSG. Under a controlled $F_t(\cdot;s_t)$8 participation profile, SubMAPL attains the largest normalized area under the coverage curve in every landscape, averaging 0.963 versus 0.924 for the strongest baseline (MA-SPL); REINFORCE collapses to 0.454 on average, and OSG plateaus below full coverage due to myopic sequential decisions.

The stress test uses random per-agent activity intervals on the Gaussian-process field. SubMAPL reaches 0.95 coverage in 5/5 scenarios with mean hitting time 580 steps versus 904 for MA-SPL, final coverage 0.999 versus 0.989, and a mean normalized-area advantage of 0.043 (95% CI [0.009, 0.078]). MA-OSEA, REINFORCE, and OSG reach 0.95 coverage in no scenario. The advantage is attributed to observation-conditioned policy tables that are restored on reactivation, avoiding transient re-adaptation. One caveat is that the learned policies are trained in a closed system and merely evaluated on open ones; the theory, by contrast, analyzes online learning along the open trajectory, so the experimental protocol does not directly exercise the regret guarantee.

Limitations and open questions

Several restrictions are acknowledged or evident. The guarantees require bounded marginal gains, monotone submodular stage utilities, and—crucially—agent arrival/departure processes independent of policies and transitions; adversarial participation dynamics are not covered. The $F_t(\cdot;s_t)$9 finite-horizon bound holds only under the time-expanded submodularity assumption with policy-independent feasible families, and the general-case bound carries an uncontrolled benchmark-mismatch term $\Omega_t$0. The algorithm is analyzed and implemented with tabular policies; extension to function approximation (e.g., neural policies) is left open, as is the treatment of richer communication constraints than the local neighbor observations used in experiments. The stagewise approach also leaves open whether long-horizon, value-aware objectives could improve the constant-factor guarantees.

Conclusion

The paper provides a coherent pipeline from problem structure to algorithm to guarantee: the PME makes submodular stage utilities compatible with factorized categorical policies; marginal-gain feedback yields unbiased PME gradients; KL-mirror updates preserve feasibility exactly through logit updates; and open policy migration plus an open-system tracking variation yield a $\Omega_t$1-approximation dynamic regret bound and a $\Omega_t$2-approximation finite-horizon bound under time-expanded submodularity. Empirically, SubMAPL outperforms submodular online-learning and policy-gradient baselines on open-system coverage, with the largest margins under strongly time-varying participation. The main open questions concern non-tabular policies, controlled mismatch terms in the general finite-horizon case, and participation dynamics coupled to agent behavior.