Set-Valued Bellman's Principle

• Set-valued Bellman’s principle is a generalization of dynamic programming where value functions are defined as sets, capturing uncertainty and multi-criteria objectives.
• It leverages set-based operators with contraction properties in the Hausdorff metric, ensuring unique invariant fixed points via recursive computation.
• The framework has practical applications in robust control, portfolio optimization, and risk-sensitive systems, utilizing techniques like parallel value iteration and vector optimization.

The set-valued Bellman’s principle generalizes classical dynamic programming by characterizing optimality and value functions in terms of sets (often subsets of $\mathbb{R}^n$ or more general lattices), instead of the traditional real-valued or vector-valued functions. This extension is critical for robust and multi-objective control, risk-sensitive finance, and dynamic systems with parameter uncertainty or non-scalar reward structures. The principle asserts the existence and uniqueness of invariant set-valued solutions—fixed points for certain set-based Bellman operators—and provides rigorous frameworks for their recursive computation and interpretation.

1. Formal Definition and Set-Valued Operators

Given a family of discounted Markov Decision Processes (MDPs) indexed by a compact (often convex, possibly polytopic or interval-bounded) set $\mathcal{C} \subset \mathbb{R}^{S \times A}$ of cost matrices, the standard Bellman operator for a fixed $C$,

$$f_C: \mathbb{R}^S \to \mathbb{R}^S,\quad (f_C(V))_s = \min_{a \in [A]} \left\{ C_{sa} + \gamma \sum_{s'} P_{s'sa} V_{s'} \right\},$$

is extended (“lifted”) to act on compact sets of value functions. Define $H(\mathbb{R}^S)$ as the collection of nonempty compact subsets of the value function space, with the set-valued Bellman operator

$$T_{\mathcal{C}}: H(\mathbb{R}^S) \to H(\mathbb{R}^S),\qquad T_{\mathcal{C}}(\mathcal{V}) = \mathrm{cl}\left\{ f_C(V) \mid C \in \mathcal{C},\ V \in \mathcal{V} \right\},$$

where $\mathrm{cl}$ denotes topological closure in $\mathbb{R}^S$ (Li et al., 2020, Li et al., 2020, Li et al., 2022).

This framework extends naturally to uncertainty in both costs and transition kernels, to multi-objective and vector-valued cost criteria, to time-varying and pathwise random settings, as well as to continuous-state spaces and stochastic processes (Kováčová et al., 2018, Visetti, 2021, İşeri et al., 2023, Cialenco et al., 29 Jun 2024).

2. Metric Structure and Contraction Properties

The set-valued Bellman operator acts on the metric space $(H(\mathbb{R}^S), d_H)$, where the Hausdorff distance is defined by

$$d_H(X, Y) = \max\left\{ \sup_{x \in X} \inf_{y \in Y} \|x - y\|_\infty,\ \sup_{y \in Y} \inf_{x \in X} \|x - y\|_\infty \right\}.$$

$(H(\mathbb{R}^S), d_H)$ is complete whenever $(\mathbb{R}^S, \|\cdot\|_\infty)$ is complete (Li et al., 2020, Li et al., 2022).

A key property is $\gamma$-contraction: for any $\mathcal{V}, \mathcal{W} \subset \mathbb{R}^S$ nonempty and compact,

$$d_H\big( T_{\mathcal{C}}(\mathcal{V}),\ T_{\mathcal{C}}(\mathcal{W}) \big) \leq \gamma\ d_H(\mathcal{V},\ \mathcal{W}),$$

where $\gamma \in (0, 1)$ is the MDP discount factor. This contraction holds both for set-based Bellman and policy evaluation operators, including in robust, nonstationary, and more general contractive dynamic programming settings (Li et al., 2020, Li et al., 2022).

Order-preservation (i.e., $$\mathcal{V} \subseteq \mathcal{W} \implies T_{\mathcal{C}}(\mathcal{V}) \subseteq T_{\mathcal{C}}(\mathcal{W})$$) is also satisfied. These properties ensure structural monotonicity and regularity vital for existence and uniqueness results.

3. Existence and Characterization of Fixed Points

By the Banach fixed-point theorem, a $\gamma$-contraction on a complete metric space admits a unique fixed point. Hence, there exists a unique compact set $\mathcal{V}^\star \in H(\mathbb{R}^S)$ such that

$$T_{\mathcal{C}}(\mathcal{V}^\star) = \mathcal{V}^\star,$$

with convergence (in $d_H$) for the sequence $\{\mathcal{V}^k\}$ defined via $\mathcal{V}^{k+1} = T_{\mathcal{C}}(\mathcal{V}^k)$ from any starting nonempty compact $\mathcal{V}^0$. This set $\mathcal{V}^\star$ can be interpreted as:

• The collection of all value functions $V_C^\star$ solving the Bellman equation for $C \in \mathcal{C}$
• The invariant (attractor) set for value-iteration processes under adversarial (or random) cost variations or uncertainties
• The tightest possible pointwise lower and upper bounds on the optimal value functions under the prescribed uncertainty (Li et al., 2020, Li et al., 2022)

If the uncertainty set is an interval box or polytope, $\mathcal{V}^\star$ takes the form of a hyper-rectangle; extremal value functions are realized at the corners of the admissible cost set (Li et al., 2020, Li et al., 2020). Table 1 organizes the fixed point characterization across key works:

Paper	Uncertainty Set	Fixed-Point Structure
(Li et al., 2020)	Cost intervals $\mathcal{C}$	$\mathcal{V}^\star$ hyper-rectangle
(Li et al., 2022)	$(C,P)$ in compact $M$	$\mathcal{S}^*$, compact, extremal elements achieved
(Li et al., 2020)	Box/Polytope $\Theta$	Interval bounds $[\underline{v},\overline{v}]$

4. Extensions: Multi-Objective, Risk, and Continuous-State problems

The set-valued Bellman principle underpins modern frameworks for multi-objective (vector-valued) costs, dynamic risk measures, and time-consistent robust control. Key directions include:

• Multi-objective optimization: The recursive value function is a set (or upper image) in an ordered lattice (e.g., via a convex cone) (Kováčová et al., 2018, Cialenco et al., 29 Jun 2024). The Bellman principle takes several forms, including infima and suprema in the partially ordered space—the Hopf-Lax set-valued formula and HJB equations arise in this setting (Visetti, 2021, İşeri et al., 2023).
• Multi-portfolio time-consistent risk measures: Acceptance and capital requirement sets are recursively computed via a set-valued Bellman equation, corresponding to backward composition of one-step conditional risk maps (Feinstein et al., 2015).
• Continuous and infinite-dimensional control: In linear-quadratic control, the Bellman/HJB equation is fundamentally set-valued, with exponentially many quadratic fixed-point solutions—only one of which yields both optimality and system stability (You et al., 4 Mar 2025). Enforcing positive-definite architectures ensures selection of the correct stabilizing solution.

5. Computation, Algorithms, and Practical Implications

Algorithmic implementation of the set-valued Bellman principle involves set-iteration schemes, often leveraging sampling in the space of uncertainties or tracking of extreme value functions (Li et al., 2020). For box or polytopic uncertainty, parallel value-iteration for each cost vertex suffices. In the convex or polyhedral case, modern vector optimization algorithms (e.g., Benson's outer-approximation) are used for set recursion steps (Feinstein et al., 2015, Kováčová et al., 2018).

Computational complexity depends on:

• The dimension of the value function (state space)
• The representation (and number of extreme points/facets) required for sets in recursive steps
• The structure (convex, polyhedral, general compact sets) of parameter or cost uncertainty

Table 2 summarizes computational approaches for common scenarios:

Scenario	Method
Interval-bounded MDP costs	Parallel value iteration at corners
Multiobjective mean-risk portfolio	Nodewise vector-optimization on event tree
Convex risk measures with transaction costs	Sequential convex VOP solvers

In practical contexts (e.g., robust control under wind uncertainty, multi-period financial portfolio optimization), the set-valued fixed point provides worst- and best-case value function bounds, and the iteration captures invariant behaviors under non-stationary or path-dependent randomness (Li et al., 2022, Kováčová et al., 2018).

6. Theoretical and Interpretative Implications

The set-valued Bellman principle unifies robust planning, distributional/multivariate control, and risk-sensitive optimization. Key implications include:

• Robustness: The unique fixed-point set represents the envelope of value functions achievable under all allowable parameter perturbations.
• Time consistency: Multi-objective dynamic risk and control problems, which fail classical scalar dynamic programming, regain a form of recursive optimality—provided the value function is interpreted set-valued and backward-constructed in the corresponding ordering lattice (Kováčová et al., 2018, Feinstein et al., 2015).
• Dynamic invariance: For general Markovian systems subject to uncertain or time-varying objectives and models, value-iteration sequences asymptotically approach the set-valued fixed point; in many cases, even when individual trajectories do not converge pointwise, the invariant set acts as a global attractor (Li et al., 2020, Li et al., 2022).
• Admissibility in continuous control: In continuous-state or operator settings, the set-valued equation manifests as a high-multiplicity solution set, with architecture-enforced admissibility essential for practical learning (You et al., 4 Mar 2025).

7. Applications and Illustrative Examples

The set-valued Bellman framework has been instantiated in numerous domains:

• Path planning and reachability: Set recursions compute maximal invariant or safe sets for discrete-time systems, path planning under dynamic obstacles, and robust safety verification (Jones et al., 2020).
• Portfolio optimization: Upper image recursions encode the entire efficient frontier under dynamic mean-risk criteria, enabling fully non-scalar time-consistent strategies (Kováčová et al., 2018).
• Robust MDPs and stochastic games: The set-valued fixed point bounds learning trajectories and Nash values under parameter-uncertainty or adversarial environments (Li et al., 2020).
• Stochastic control with multi-loss criteria: Bellman recursion in partially ordered lattices enables rigorous robust and vector-valued dynamic programming in time-inconsistent settings (Cialenco et al., 29 Jun 2024).
• Set-valued HJB equations: In continuous time, set-valued Hamilton-Jacobi equations with well-posedness theory and set-valued Itô calculus capture multiobjective and time-inconsistent stochastic optimization with moving scalarizations (İşeri et al., 2023, Visetti, 2021).

Applications are driven by the ability to formally guarantee extremal bounds, provide certificates under model/modeling uncertainty, and capture all admissible optimal responses across a given uncertainty set.

---

The set-valued Bellman’s principle is a rigorous and flexible extension of the classical Bellman principle, providing the mathematical and algorithmic foundation for dynamic programming with uncertainty, risk, and multi-criteria objectives. It elevates the value function to a lattice- or set-valued object, ensures unique invariant fixed points in the space of compact value sets, and subsumes robust, stochastic, and multiobjective dynamic optimization under a unified contractive fixed-point framework (Li et al., 2020, Li et al., 2022, Kováčová et al., 2018, You et al., 4 Mar 2025, İşeri et al., 2023, Cialenco et al., 29 Jun 2024, Feinstein et al., 2015, Visetti, 2021, Jones et al., 2020, Li et al., 2020).