---
title: Submodular Rewards in RL and Optimization
url: https://www.emergentmind.com/topics/submodular-rewards
type: topic
---

# Submodular Rewards in RL and Optimization

Submodular rewards arise when the value of a set—of actions, states, or visited elements—exhibits the diminishing returns property: the marginal increase from adding an element to a set decreases as the context set grows. In sequential decision-making, planning, reinforcement learning (RL), bandit optimization, multi-agent systems, and subset recommendation, submodular rewards define a rich class of objectives extending beyond additive (modular) rewards. Modeling rewards as monotone submodular functions captures diverse application domains such as exploration, experimental design, influence maximization, sensor placement, and coverage tasks.

## 1. Formal Definition and Properties

Let $\Omega$ be a finite ground set, commonly realized as the set of possible state–action pairs, edges in a graph, or items in a recommendation domain. A reward function $f:2^\Omega\rightarrow\mathbb{R}_+$ is termed **submodular** if for all $S\subseteq T\subseteq\Omega$ and $e\notin T$,
$$
f(S \cup \{e\}) - f(S) \geq f(T \cup \{e\}) - f(T),
$$
expressing diminishing marginal returns. Equivalently, for all $S,T\subseteq \Omega$,
$$
f(S) + f(T) \geq f(S \cup T) + f(S \cap T).
$$
Frequently, rewards are assumed monotone: $S\subseteq T \implies f(S)\le f(T)$. These properties yield powerful structural guarantees for optimization and are central to approximation algorithms in both offline combinatorial settings and sequential decision-making domains such as MDPs, RL, and multi-agent planning [2010.11863][2407.09905][2307.13372].

## 2. Submodular Rewards in Sequential Decision Processes

In planning and RL, submodular rewards assign value not to individual states or actions, but to the set of state–action pairs, states, or visited indices along an agent's trajectory. The general model consists of:
- A finite-horizon MDP/MDP tuple $M=(S, A, P, \mu, H)$.
- Trajectories $T=(s_1,a_1,\dots,s_H,a_H)$ leading to a set $S_T$ (e.g., state–action pairs $(s_h,a_h)$ or visited states).
- Cumulative reward is $f(S_T)$, with $f$ monotone submodular.

The agent's objective is to optimize $J(\pi) := \mathbb{E}[f(S_T)\mid \pi]$ over policies $\pi$, which is non-additive and generally history-dependent. This framework includes as special cases the classical additive reward ($f$ modular) and subset maximization under cardinality constraints ($H=k$, $|A|=n$) [2010.11863][2307.13372].

## 3. Optimization Methodologies and Algorithmic Frameworks

### 3.1 Multilinear Extension and Continuous Greedy

To address non-additive, non-Markovian submodular objectives, [2010.11863] employs the **multilinear extension** $F:[0,1]^{\Omega}\rightarrow\mathbb{R}_+$ of $f$:
$$
F(x) = \mathbb{E}[f(S^x)] = \sum_{S\subset \Omega} f(S)\prod_{e\in S}x_e\prod_{e\notin S}(1-x_e),
$$
where $S^x$ is a random set including each $e$ independently with probability $x_e$.

A continuous-greedy ascent constructs solutions in the space of marginals $x\in [0,1]^{\Omega}$, iteratively updating along the direction maximizing $\nabla F$. Discretized updates rely on approximated gradients via sampling and repeated dynamic-programming MDP solves using the surrogate immediate reward $w_t(e)\approx \mathbb{E}[f(S^{y_{t-1} \cup \{e\}})-f(S^{y_{t-1}})]$. The final randomized policy (or a pipage-rounded deterministic policy) yields strong approximation ratios [2010.11863].

### 3.2 Semi-gradient Surrogate Approaches

Global RL methods [2407.09905] adopt an iterative **submodular semi-gradient** scheme. At each iteration, a tight modular (additive) lower-bound (subgradient) $m_{X}^{\sigma}$ of $F$ at the current set $X$ is constructed:
$$
m_X^\sigma(v) = F(S_i^\sigma) - F(S_{i-1}^\sigma),
$$
with a permutation $\sigma$ starting with $X$. Maximizing this modular surrogate via standard RL or DP yields new policies. Ascent in $F$ is monotonic, and curvature-dependent approximation ratios are available.

### 3.3 Greedy, Augmented-Greedy, and Bandit Algorithms

In stochastic and bandit submodular optimization, extensions of greedy, double-greedy, or "randomized greedy learning" approaches [2302.01324][2410.08578][2310.18465] are employed, frequently achieving $\frac12$ or $1-1/e$ approximation ratios (under monotonicity). Regret bounds in the stochastic bandit and combinatorial bandit paradigms explicitly leverage submodularity to reduce exploration complexity.

### 3.4 Policy Gradient and Pruned Submodularity Graphs

Policy gradient algorithms adapted to submodular reward domains exploit marginal gains in trajectory elements. For example, in SubPO [2307.13372], the policy is optimized via stochastic/natural gradients of $J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[f(\tau)]$. Pruned submodularity graphs (as in [2507.13834]) sparsify the search over state–action pairs by identifying high-divergence (high marginal-gain) states, improving training stability and sample efficiency.

## 4. Theoretical Guarantees and Curvature-Dependent Bounds

Performance guarantees for submodular reward maximization are often parametrized by the submodular curvature $\kappa$. For a monotone submodular function $f$,
$$
\kappa = 1 - \min_{e\in\Omega} \frac{f(\Omega) - f(\Omega\setminus \{e\})}{f(\{e\})},
$$
where $\kappa=0$ indicates a modular function. Approximation ratios degrade gracefully with increasing curvature:
- Greedy or continuous-greedy achieves $1/(p+\kappa)$ or $(1-1/e)$ under $p$-system constraints [1209.3759][2010.11863].
- For global RL, after one semi-gradient step, $J(\pi_1)\ge (1-\kappa)J(\pi^*)$ [2407.09905].
- In the general setting, robust regret bounds interpolate between $T^{2/3}$ (limited horizon) and $\sqrt{T}$ (long-horizon) regimes, with submodularity enabling significant exploration complexity reduction compared to naïve arms-as-subsets approaches [2310.18465][2410.08578].

## 5. Applications and Empirical Evaluation

Submodular rewards underpin a wide spectrum of sequential decision domains:
- **Exploration and coverage:** Maximizing state visit diversity [2407.09905][2307.13372][2507.13834], area cover (robot exploration, vision coverage).
- **Experimental design/informative path planning:** D-optimality, mutual information objectives [2010.11863][2407.09905][2307.13372].
- **Multi-agent coordination:** Decentralized task assignment under capacity and exclusivity via submodular partition matroids [2212.14624].
- **Bandit subset selection and combinatorial optimization:** Influence maximization, recommendation, contextual and full-bandit feedback [2302.01324][2112.02165][2310.18465][2410.08578].
- **Adaptive and nonmonotone regimes:** Revenue maximization with costs, stochastic influence/truncation constraints [2103.00384][2109.15180].

Empirical results across RL and planning benchmarks, graph bandits, recommendation, and multi-agent VRP demonstrate that submodular-aware methods consistently outperform greedy modular baselines, directly exploiting diminishing returns to enhance sample efficiency and policy performance [2010.11863][2407.09905][2307.13372][2507.13834][2103.00384][2212.14624][2305.02806].

## 6. Limitations, Hardness, and Open Problems

Despite their favorable structure, most submodular reward maximization problems remain NP-hard, and strong inapproximability results apply even in deterministic tabular domains [2307.13372][2507.13834][2407.09905]. Stochastic and bandit settings amplify hardness due to noisy feedback and partial information. No polynomial-time algorithm can, in general, achieve a constant-factor approximation better than $(1-\kappa)$ for curvature $\kappa<1$ (unless $P=NP$). In the adaptive regularization and partition-constrained settings, nonmonotonicity and negative values require specialized algorithmic reductions and surrogate objective formulations for effective optimization [2103.00384][2109.15180].

A plausible implication is that further algorithmic progress will depend on leveraging additional domain structure, exploiting low-curvature functions, or developing scalable sampling and surrogate estimation techniques.

## 7. Connections and Broader Impact

Submodular reward modeling bridges the gap between additive-reward RL and combinatorial submodular maximization, supporting complex objectives involving coverage, diversity, information, risk, and interaction effects. This modeling paradigm is foundational in modern RL, multi-agent cooperation, adaptive experiment design, and recommendation under fairness/bias constraints [2305.02806]. Future research directions include improved practical algorithms for non-monotone/non-submodular objectives, tighter curvature-dependent analysis, and domain-adaptive neural surrogates for large-scale, high-dimensional deployments.

Source: https://www.emergentmind.com/topics/submodular-rewards