Discretized Dynamic Programming Methods

• Discretized dynamic programming is a numerical approach that replaces continuous domains with finite grids to solve complex Bellman recursions.
• It employs uniform grids, dual transformations, and tree-structured methods to manage error and reduce computational complexity.
• The method is widely used in stochastic control, MDPs, and high-dimensional optimization, offering rigorous error bounds and scalability.

A discretized dynamic programming (DP) approach refers to a family of numerical methods for solving dynamic programming problems by replacing continuous state, action, or constraint domains with finite grids or discrete representations. This discretization transforms analytically or numerically intractable infinite-dimensional optimization or Bellman equations into finite-dimensional, algorithmically solvable recursions. Discretization is fundamental in stochastic control, optimal control, Markov decision processes (MDPs), and related areas, encompassing uniform-grid schemes, tree-structured discretizations, dual/Legendre-transformed methods, pruning algorithms, and error-control techniques. Rigorous error bounds often accompany these methods, quantifying the impact of discretization on bias and optimality.

1. Formalization of Discretized Dynamic Programming

Dynamic programming for discrete-time stochastic optimal control typically seeks the value function

$$V_k(x_k) = \inf_{\pi} \mathbb{E}^{\pi} \left[ \sum_{j=k}^{N-1} c(x_j, u_j) \right]$$

subject to dynamics, constraints, and possibly other dependence, often in the form of Bellman recursions. Discretization arises when some domain (state, action, budget, risk threshold, measure) is continuous, infinite, or high-dimensional, prohibiting direct solution by tabulation or exact finite recursion. The main idea is to construct finite sets or grids for these domains, replacing integrals, minimizations, or infima with combinatorial operations. For example:

• Uniform grid discretization of a continuous risk threshold domain $[R_-, R_+]$, replacing $r_k \in [R_-, R_+]$ by a grid $R_k^{(\tau)}$ of step size $\Delta$ (Chow et al., 2015).
• Hash-table and quantized key representation in belief/MOPDPs, where continuous beliefs $b\in\Delta(\mathcal{S})$ are replaced by $$(\lceil D b(s_1)\rceil, \ldots, \lceil D b(s_{|S|})\rceil)$$ (Adalgeirsson et al., 2022).
• Grid-based finite partitioning in high-dimensional $\mathbb{Z}^n$ state/action space (Lebedev et al., 2020).
• Tree-structured enumeration of time-discretized reachable states, eliminating fixed spatial interpolation (Alla et al., 2018).

Discretization transforms dynamic programming into finite-dimensional nonlinear programming, facilitating algorithmic solution, value-iteration, policy-iteration, or specialized recursions.

2. Uniform-Grid Schemes with Error Control

The canonical use case is the discretization of continuous variables in the Bellman recursion. For instance, risk-constrained stochastic control introduces a continuous risk threshold $r_k$ in the value function domain, rendering standard finite-state DP methods inapplicable. The solution is to impose a uniform grid

$$\{ r_k^{(0)} = R_-, r_k^{(1)}, \ldots, r_k^{(t+1)} = R_+ + \epsilon \}, \quad \Delta = (R_+ - R_-)/t$$

and define a finite, discrete approximation to the Bellman operator:

$$T^\Delta_k[V](x_k, r_k^{(\tau)}) = \min_{(u, r^{D, \prime}) \in F^D_k(x_k, r_k^{(\tau)})} \left\{ c(x_k, u) + \sum_{x'} Q(x'|x_k, u) V(x', r^{D, \prime}(x')) \right\}$$

where $F^D_k$ enumerates only grid points. One proves that, under mild Lipschitz assumptions on costs and mappings, the sup-norm error between the true and discretized value function is linear in the grid size:

$\| V^\Delta_k - V_k \|_\infty \leq C \Delta,$

with $C$ determined by the sum of Lipschitz constants over all stages. The implementation involves a nested loop over state, grid point, control, and grid tuple assignments, with feasibility pruning for constraints (Chow et al., 2015).

Discretization Domain	Grid Type	Error Order	Reference
Risk threshold $r_k$	Uniform	$O(\Delta)$	(Chow et al., 2015)
State, action	Multivariate grid	$O(\max\{\delta_x, \delta_u\})$	(Kolarijani et al., 2020)
Belief simplex	Quantized hash	Empirical, $O(1/D)$	(Adalgeirsson et al., 2022)

3. Structure-Preserving and Dualized Discretization

For input-affine discrete-time systems, computational complexity of standard discretized DP is prohibitive ($O(|\Xh||\Uh|)$ per Bellman update). By exploiting affine dynamics and separability in cost, one can dualize the Bellman update via discrete Legendre–Fenchel transforms:

$$(\mathcal{T} J)(x) = \sup_{y \in \mathcal{Y}} \left\{ \langle y, s(x) \rangle - [C_x^*(-i(x)y) + J^*(y)] \right\}$$

where discrete conjugation replaces minimization by maximization/addition over dual grids. For separable problems $C(x, u) = s(x) + i(u), f(x, u) = s(x) + Bu$, the update further factorizes, reducing total complexity to $O(|\Xh|)$ versus $O(|\Xh||\Uh|)$. Error analysis quantifies the approximation in terms of grid diameters and dual grid coverage (Kolarijani et al., 2020).

4. Algorithms for High-Dimensional or Non-Standard Domains

In very high-dimensional settings (state spaces $\mathbb{Z}^n$ with $n \gtrsim 10$), even grid-based discretization is infeasible. Structure in value functions such as submodularity and concave extensibility enables construction of outer and inner bounds via adaptive hyperplane approximations:

$$Q_t^i(x) = \min_{j=0,\ldots,i} H^j_t(x), \quad H^j_t(x) = a^j_t \cdot x + b^j_t$$

The DP is approximated above by a piecewise affine function, with stochastic forward sweeps generating lower bounds and backward hyperplane fitting propagating deterministic upper bounds. Under the stated assumptions, the bounds converge to the true value function, and the method is guaranteed to terminate after at most $|\mathcal{T}||\mathcal{X}|$ iterations (Lebedev et al., 2020).

In continuous-control or hybrid domains, tree-structured approaches by aligning discretization strictly with the transition graph eliminate the need to interpolate or build static grids, supporting problems with hundreds to thousands of state dimensions. Pruning and merging based on the local Lipschitz constant control the explosion in tree size without compromising convergence (Alla et al., 2018).

5. Discretization in Non-Markovian, Risk, and Measure-Valued DPs

Discretized DP arises in diverse non-classical contexts:

• In distribution-constrained control (McKean–Vlasov problems), discretization occurs in the infinite-dimensional space of probability measures. Although Bellman recursions hold formally on $\mathcal{P}_2(E)$, practical DP requires parametric or grid approximations, recasting the infinite-dimensional problem as finite via moment mappings or quantized measure supports (Pham et al., 2015).
• In partially observable or belief-space MDPs (POMDPs), discretization of the belief simplex via quantization (e.g., $D$-bin hash keys) underlies value-iteration and real-time DP methods, enabling storage and updating of value functions in hash tables, without explicit grid over the simplex (Adalgeirsson et al., 2022).
• In evaluating probabilistic constraints (e.g., Gaussian integrals over polytopes), DP and discretization enable the conversion of a continuous high-dimensional integral into a sequence of expected value computations over a finite number of grid points, with explicit error bounds in terms of smoothing and grid partition parameters (Jones et al., 2018).

6. Complexity, Implementation and Extensions

The computational effort of discretized DP is dominated by the cardinality of the discretized domains. For methods involving full grid enumeration, complexity grows exponentially in the number of discretized variables (curse of dimensionality). Advances include:

• Pruning infeasible or sub-optimal grid configurations early ("branch & bound") (Chow et al., 2015).
• Randomized or variable-resolution grids concentrating discretization in high-curvature regions or frequent trajectories (Chow et al., 2015).
• Hyperplane-based or dual-grid approaches reducing complexity from quadratic to near-linear in the number of states (Kolarijani et al., 2020, Lebedev et al., 2020).
• Memory scaling with the number of visited grid/belief keys, supporting anytime algorithms with explicit trade-offs between error bound and computational resources (Adalgeirsson et al., 2022).
• Tree-based data structures and context-specific merging reducing complexity below the full grid baseline (Alla et al., 2018).

Algorithmic extensions span randomized grid sampling, function approximation (basis/extensions in continuous domains), and on-the-fly reinforcement learning with discretized Bellman operators (Chow et al., 2015, Adalgeirsson et al., 2022).

7. Representative Applications and Empirical Performance

Discretized dynamic programming underlies a range of applied and theoretical domains:

• Risk-constrained stochastic optimal control, enabling finite-approximate enforcement of continuous risk budgets with proven $O(\Delta)$ error (Chow et al., 2015).
• Input-affine optimal control with separable cost structures, where direct dualization slashes per-step DP complexity and yields sharp error formulations (Kolarijani et al., 2020).
• High-dimensional resource allocation, e.g., delivery-slot pricing, where submodular/concave properties permit scalable approximation schemes with provable finite convergence (Lebedev et al., 2020).
• Belief-space and POMDP planning, where quantization and bounded (upper/lower) DP enable improved anytime performance over state-of-the-art continuous or point-based solvers (Adalgeirsson et al., 2022).
• Tree-structured DP for high-dimensional PDE control, eliminating spatial interpolation and grid memory with first-order accuracy and feasible scaling to $\mathcal{O}(10^3)$ state dimensions (Alla et al., 2018).

Empirical demonstrations validate the mathematical error bounds. For example, in two-dimensional Gaussian-integral evaluations, reducing the grid spacing by factor 2 halves the numerical error as per theory (Jones et al., 2018). In belief-branch RTDP, increasing belief discretization sharpens value approximations with commensurate growth in computation, as shown by average discounted reward and planning time comparisons (Adalgeirsson et al., 2022). Tree pruning turns otherwise infeasible full-tree computations into tractable high-dimensional control policies (Alla et al., 2018).

The discretized dynamic programming paradigm remains central in computational optimal control, enabling rigorous performance characterization and scalable computation for highly structured or high-dimensional dynamic systems across stochastic, risk-aware, and partially observed settings.