Bellman–ReinMax Operator

• Bellman–ReinMax operator is a nonlinear contraction mapping in reinforcement learning that targets the maximum reward along a trajectory rather than the cumulative sum.
• It replaces the additive Bellman update with a max-based formulation, leading to unique algorithmic behavior and risk-seeking policies in practical applications.
• This operator demonstrates fixed-point convergence and finds applications in areas such as de novo molecular design and controlled jump-diffusion processes.

The Bellman–ReinMax operator is a nonlinear contraction mapping arising in reinforcement learning (RL) for maximizing the expected maximum reward encountered along a trajectory, departing fundamentally from the classical Bellman operator which targets the expected sum of discounted rewards. It provides the dynamic programming backbone for a class of maximum-reward objectives relevant in domains such as de novo molecular design, where the goal is to uncover rare, high-reward states rather than optimizing cumulative return (Gottipati et al., 2020). The Bellman–ReinMax paradigm also stands as a canonical example of non-additive dynamic programming, and its analysis elucidates distinct algorithmic and theoretical properties compared to conventional Bellman operators.

1. Formulation and Mathematical Structure

Let $(S, A, P, r, \gamma)$ denote a Markov decision process (MDP) with state space $S$, action space $A$, transition kernel $P(s'|s,a)$, immediate reward $r(s,a)$, and discount factor $\gamma \in [0,1)$. Instead of the standard RL objective, which seeks a policy $\pi$ maximizing the expected sum of discounted rewards, the Bellman–ReinMax framework defines the ReinMax–Q-function for a policy $\pi$: $$Q^{\pi}_{\max}(s,a) \coloneqq \mathbb{E}_{s'\sim P(\cdot|s,a),\,a'\sim\pi(\cdot|s')}\Big[\max\{ r(s,a),\, \gamma\, Q^{\pi}_{\max}(s',a') \}\Big]$$ and the optimal value function as $$Q^{\star}_{\max}(s,a) = \max_{\pi} Q^{\pi}_{\max}(s,a)$$. The Bellman–ReinMax operator is then

$$(\mathcal{T}_{\rm ReinMax} Q)(s,a) \coloneqq \max\left\{ r(s,a),\, \gamma\, \mathbb{E}_{s'\sim P(\cdot|s,a)} \left[ \max_{a'} Q(s',a') \right] \right\}$$

and for policy evaluation,

$$(\mathcal{T}^{\pi}_{\rm ReinMax} Q)(s,a) \coloneqq \mathbb{E}_{s'\sim P(\cdot|s,a),\,a'\sim\pi(\cdot|s')} \left[ \max\{ r(s,a),\, \gamma\, Q(s',a') \} \right].$$

Both admit fixed-point characterizations: $Q^{\star}_{\max}$ is the unique fixed point of $\mathcal{T}_{\rm ReinMax}$; $Q^\pi_{\max}$ for $\mathcal{T}_{\rm ReinMax}^\pi$. This structure differs essentially from the classical Bellman operator, which is additive in the immediate and future rewards.

2. Fixed-Point Theory and Contraction Properties

A defining property of the Bellman–ReinMax operator is that it is a $\gamma$-contraction in the supremum norm: $$\| \mathcal{T}_{\rm ReinMax} Q_1 - \mathcal{T}_{\rm ReinMax} Q_2 \|_\infty \le \gamma \| Q_1 - Q_2 \|_\infty$$ for any bounded $Q_1,Q_2: S \times A \to \mathbb{R}$. This is due to the fact that both the maximization map $x \mapsto \max\{r,x\}$ and the maximization over actions are non-expansive in $\ell^\infty$, and the transition expectation is itself non-expansive. Thus, by Banach’s fixed-point theorem, repeated application of $\mathcal{T}_{\rm ReinMax}$ converges at a geometric rate to the unique fixed point, $Q^{\star}_{\max}$, from any initialization $Q_0$. The same holds for the policy evaluation operator.

3. Algorithmic Realizations

ReinMax Value Iteration

The dynamic programming recursion is: $$Q_{k+1}(s,a) = \max\{ r(s,a),\, \gamma\, \sum_{s'} P(s'|s,a) \max_{a'} Q_k(s',a') \}.$$ This procedure is analogous to standard value iteration for MDPs, but with a non-additive “max” in place of a sum.

ReinMax Q-learning

In the tabular case, with transition sample $(s_t, a_t, r_t, s_{t+1})$, the update is: $$Q(s_t, a_t) \gets Q(s_t, a_t) + \alpha_t \left( \max\{ r_t,\, \gamma \max_{a'} Q(s_{t+1}, a') \} - Q(s_t, a_t) \right)$$ Under standard Robbins–Monro conditions and sufficient exploration, this update converges to $Q^{\star}_{\max}$ due to the contraction property (Gottipati et al., 2020).

4. Comparison to Classical Bellman Operators

In the conventional (additive) Bellman operator, the update is

$$(\mathcal{B}Q)(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s'} \left[ \max_{a'} Q(s',a') \right]$$

whereas the Bellman–ReinMax form replaces summation by maximization: $$(\mathcal{T}_{\rm ReinMax}Q)(s,a) = \max\{ r(s,a),\, \gamma\, \mathbb{E}_{s'} [\max_{a'} Q(s', a')] \}$$ The key differences are:

• Non-additivity: The reward update in Bellman–ReinMax is a “max” instead of “+”.
• Contraction: Both are $\gamma$-contractions in sup-norm.
• Optimal Policy: Bellman–ReinMax yields a policy that maximizes the expected maximum one-step reward along a trajectory; the classical Bellman operator maximizes the cumulative sum.

A plausible implication is that Bellman–ReinMax policies may exhibit risk-seeking behavior, prioritizing pathways to rare, large rewards over accumulations of smaller ones.

5. Empirical and Practical Insights

Empirical results in benchmark settings illustrate the behavioral divergence between the two operators:

• In a “gold-mining” grid world, classical Q-learning seeks to maximize aggregate gold, while ReinMax value iteration seeks routes leading to the single highest-value mine, even at the expense of smaller early gains.
• In molecular design tasks, e.g., PGFS+MB for HIV activity, using the ReinMax TD target ($\max\{r, \gamma Q\}$) discovers higher-scoring candidate molecules. The agent is not “distracted” by high average cumulative returns that can preclude discovery of rare, high-reward molecules (Gottipati et al., 2020).

6. Relationship with Softmax and Other Generalized Bellman Operators

The Bellman–ReinMax operator is closely related to max-type operators such as $T_{\max}$ and tunable “soft” generalizations: $$(T_{\rm soft}Q)(s,a) = r(s,a) + \gamma \sum_{s'} P(s'|s,a) \sum_{a'} \frac{e^{\beta Q(s',a')}}{\sum_b e^{\beta Q(s',b)}} Q(s',a')$$ As $\beta \to \infty$, $T_{\rm soft} \to T_{\max}$, converging to the classical Bellman–optimality operator (Song et al., 2018). Unlike the Bellman–ReinMax operator, $T_{\rm soft}$ is not guaranteed to be a contraction, but the fixed point approaches the Bellman–optimal value as $\beta$ increases. Each operator forms a spectrum with distinct bias-variance and overestimation properties.

Operator	Bellman-optimal?	Direct Policy?	Contraction ($\gamma$)?
Max-Bellman	yes	greedy	yes
ReinMax	no	max-of-sequence	yes
Softmax-Bellman	no	Boltzmann	usually not
Mellowmax	no	no (requires root)	yes

7. Connections to Nonlocal Bellman Operators in PDE Theory

In PDE contexts, Bellman-type operators also manifest as nonlinear nonlocal maps, for instance as the generator of controlled jump-diffusion processes (Dai et al., 2020). Here, the uniformly elliptic nonlocal Bellman operator is defined as

$$Iu(x) = \inf_{\alpha \in A} \text{p.v.} \int_{\mathbb{R}^n} [u(x+y)-u(x)]\, K_\alpha(y) dy$$

with $K_\alpha$ subject to ellipticity and symmetry bounds, unifying a wide range of control-theoretic and geometric nonlocal equations. While structurally distinct from the Bellman–ReinMax operator of RL, the unifying theme is the optimality principle applied to either the maximum over actions or parameters, highlighting the breadth of the Bellman paradigm across discrete and continuous stochastic control.

---

The Bellman–ReinMax operator introduces a non-additive, max-based dynamic programming recursion suitable for RL tasks targeting rare, high-value outcomes rather than cumulative reward. It preserves contraction and fixed-point properties, but fundamentally alters the nature of optimal policies, with proven advantages in practical applications such as drug discovery where identifying exceptional actions or states is paramount (Gottipati et al., 2020).