---
title: Shapley-Bellman Backward Recursion
url: https://www.emergentmind.com/topics/shapley-bellman-backward-recursion
type: topic
---

# Shapley-Bellman Backward Recursion

Shapley-Bellman backward recursion refers to the process by which the output of a feed-forward ReLU neural network is retrieved as the saddle-point value of a zero-sum, turn-based, stopping game known as the ReLU-net game. In this framework, each layer and neuron of the network corresponds to a state in the game, with the network’s input interpreted as a terminal reward and the feed-forward calculation expressed as a backward recursion over the game's structure. The recursion relies on coordinate-wise maximization and minimization (the Shapley equations), akin to dynamic programming in turn-based zero-sum games, and generalizes to other activation functions such as Softplus via entropic regularization. This approach enables alternative perspectives on evaluation, verification, and learning in neural networks through game-theoretic constructs, path-integral representation, and monotonicity arguments [2512.20582].

## 1. Construction of the ReLU-Net Game

A ReLU network of depth $L$ is associated with a zero-sum, turn-based, stopping game with horizon $L+1$. Each neuron $(\ell,i)$ in layer $\ell$ results in two game states: $(\ell,i,+)$ (“Max’s turn”) and $(\ell,i,-)$ (“Min’s turn”). There is also a cemetery absorbing state $\perp$ with zero reward. The player whose turn it is chooses either to stop (transitioning immediately to $\perp$) or to continue to the next layer. Transitions are defined probabilistically using the weight matrices $W^\ell \in \mathbb{R}^{k_\ell\times k_{\ell+1}}$ and a "discount factor" $\gamma^\ell_i = \sum_j|W^\ell_{i,j}|$. The transition probability from $(\ell,i,\epsilon)$ to $(\ell+1,j,\eta)$ is determined by the sign-decomposition of $W^\ell_{i,j}$ and normalization by $\gamma^\ell_i$, specifically:

- $P^\ell_{i\epsilon, j\eta} = (W^\ell_{i,j})^+ / \gamma^\ell_i$ for $(\epsilon,\eta) = (+,+)$ or $(-,-)$,
- $P^\ell_{i\epsilon, j\eta} = (W^\ell_{i,j})^- / \gamma^\ell_i$ for $(\epsilon,\eta) = (+,-)$ or $(-,+)$,
- where $(a)^+ = \max(a,0)$ and $(a)^- = \max(-a,0)$.

Immediate rewards are $b^\ell_i$ at $(\ell,i,+)$ and $-b^\ell_i$ at $(\ell,i,-)$, with terminal rewards at layer $L+1$: $(L+1,j,+)$ gets $x_j$, $(L+1,j,-)$ gets $-x_j$ [2512.20582].

## 2. Shapley–Bellman Backward Recursion Equations

The game value is computed via Shapley–Bellman backward recursion. Define $V^\ell_{i+}(x)$ (Max’s value) and $V^\ell_{i-}(x)$ (Min’s value) for state $(\ell,i,+)$ and $(\ell,i,-)$ respectively:

- Terminal condition: $V^{L+1}_{j+}(x)= x_j$, $V^{L+1}_{j-}(x)= -x_j$
- Recursive equations for $\ell=L,\dots,1$:
  $$
  V^\ell_{i+}(x) = \max \left\{ 0,\ \gamma^\ell_i [ \sum_{j:W^\ell_{i,j}>0}P^\ell_{i+,j+}V^{\ell+1}_{j+}(x) + \sum_{j:W^\ell_{i,j}<0}P^\ell_{i+,j-}V^{\ell+1}_{j-}(x) ] + b^\ell_i \right\}
  $$
  $$
  V^\ell_{i-}(x) = \min \left\{ 0,\ \gamma^\ell_i [ \sum_{j:W^\ell_{i,j}>0}P^\ell_{i-,j-}V^{\ell+1}_{j-}(x) + \sum_{j:W^\ell_{i,j}<0}P^\ell_{i-,j+}V^{\ell+1}_{j+}(x) ] - b^\ell_i \right\}
  $$
By induction, $V^\ell_{i+}(x) = y^\ell_i$ and $V^\ell_{i-}(x) = -y^\ell_i$, where $y^\ell_i$ is the feed-forward network output at neuron $(\ell,i)$ [2512.20582]. The max/min recursion directly reproduces the coordinate-wise ReLU.

## 3. Discrete Feynman–Kac Path-Integral Formula

The game’s output admits a discrete path-integral (Feynman–Kac-type) representation. Given any pair of policies $\pi$ (Max) and $\sigma$ (Min), a trajectory $\alpha$ initiates at $(\ell,i,+)$ and evolves according to chosen actions and transition probabilities. For each $\alpha$,
- Transition probability: $P^{\pi,\sigma}(\alpha) = \prod_{t=\ell}^{\ell+k-1}P^t_{\alpha(t),\alpha(t+1)}$
- Reward aggregate:
  $$
  r(\alpha) = \sum_{t=\ell}^{\ell+k-1} \left( \prod_{u=\ell}^{t-1}\gamma^u \right) r(\alpha(t)) + \left( \prod_{u=\ell}^{\ell+k-1}\gamma^u \right) \varphi(\alpha(\ell+k))
  $$
with the game value expressed as
$$
V^{\ell,\pi,\sigma}_{i+}(x) = \sum_{\alpha}P^{\pi,\sigma}(\alpha)r(\alpha).
$$
The saddle-point $V^\ell_{i+}(x) = \max_\pi \min_\sigma V^{\ell,\pi,\sigma}_{i+}(x)$, and at optimal policies $(\pi^*,\sigma^*)$, $y^\ell_i = V^\ell_{i+}(x)$ [2512.20582]. This duality connects dynamic programming/backward recursion and expectation/path-integral perspectives.

## 4. Monotonicity, Robustness Bounds, and Certification

Replacing the terminal map $x \mapsto (\pm x)$ with $\varphi$ generalizes the recursion, maintaining coordinate-wise monotonicity:
- If $x_{\min} \leq x \leq x_{\max}$ pointwise, then
  $$
  y = V^1(x,-x) \in [ V^1(x_{\min},-x_{\max}),\ V^1(x_{\max},-x_{\min}) ]
  $$
- This establishes explicit bounds on outputs from input bounds.

For fixed policies, the following piecewise-linear bounds arise:
- Lower bound: $f^\pi(x) = \min_\sigma V^{1,\pi,\sigma}_{1+}(x)$ is concave and piecewise-linear; its super-level set $\{x\mid f^\pi(x)\geq\alpha\}$ is a polyhedral certificate of $y(x)\geq\alpha$.
- Upper bound: For $\sigma$ fixed, $g^\sigma(x)$ is convex/PL; its sub-level set certifies $y(x)\leq\beta$.

This mechanism provides robustness certification against input perturbations and adversarial scenarios [2512.20582].

## 5. Inverse Game Formulation for Network Learning

Under supervised learning, given sample pairs $\{(x^{(m)},y^{(m)})\}$, ReLU-net game training is reformulated as an "inverse game" problem. The unknowns are transition probabilities $P^\ell$ and rewards $b^\ell$, constrained to reflect the network weights and normalization. The Shapley equations become equality constraints:
$$
V^\ell_{i+}(x^{(m)};P,b) = y^{\ell,(m)}_i, \hspace{1cm} \forall\ \ell, i, m
$$
subject to
$$
P^\ell_{i\epsilon, \cdot} \geq 0, \quad \sum_{j,\eta} P^\ell_{i\epsilon, j\eta} = 1,
$$
with $P^\ell_{(i,\pm),(j,\pm)} \propto (W^\ell_{i,j})^\pm$. This yields a feasibility or least-squares optimization:
$$
\min_{P, b} \sum_m \| y^{(m)} - V^1(x^{(m)}; P, b) \|^2
$$
subject to the transition/reward constraints [2512.20582]. This framework interprets neural network learning as identification of game structure from observed outputs.

## 6. Entropic Regularization and Extension to Softplus Networks

For neural networks with Softplus activation, the game-theoretic model is modified via entropic regularization. At state $s=(\ell,i,\epsilon)$, the reward is augmented by $-\tau\log p$ (for Max, with $p$ probability of "continue") or $+\tau\log q$ (for Min). The recursion then yields:
- For Max:
  $$
  V^\ell_{i+,\tau}(x)=\tau\log\left[1 + \exp\left(\frac{\gamma^\ell_i\sum_j \text{sign-split}P^\ell V^{\ell+1}_{j,\cdot} + b^\ell_i}{\tau}\right)\right]
  $$
- For Min:
  $$
  V^\ell_{i-,\tau}(x)= -\tau\log\left[1 + \exp\left(-\frac{\gamma^\ell_i\sum_j \text{sign-split}P^\ell V^{\ell+1}_{j,\cdot} - b^\ell_i}{\tau}\right)\right]
  $$
which reproduces the Softplus activation, $\text{softplus}_\tau(z)=\tau \log(1+e^{z/\tau})$. The path-integral formula generalizes to a log–sum–exp over all trajectories, representing the free-energy interpretation of the output [2512.20582]. This suggests a fundamental equivalence between neural activation smoothing and entropy-regularized stopping games.

## 7. Connections, Implications, and Future Directions

The Shapley–Bellman backward recursion in the context of neural networks provides a rigorous bridge between dynamic programming, game theory, and deep learning architectures. It enables path-integral interpretations for network outputs, monotonicity-based verification, robustness analysis, and conceptualizes neural training as a constrained inverse game identification. The entropic regularization mechanism elucidates the link between classic ReLU activations and smooth variants such as Softplus. A plausible implication is a broader class of game-theoretic neural architectures in which backward recursion and optimal policy pairs systematically determine both evaluation and training regimes, potentially extending robustness and verification capabilities across a wider array of network types [2512.20582].

Source: https://www.emergentmind.com/topics/shapley-bellman-backward-recursion