Approximate Dynamic Programming Methods

• ADP is a set of methods that approximate solutions to complex Markov Decision Processes by replacing exact cost-to-go functions with parametrized surrogates.
• It reformulates classical Bellman equations into optimization problems such as bilinear programs or MILPs, providing robust error bounds and performance guarantees.
• ADP techniques offer practical scalability and have shown empirical success in control tasks like the inverted pendulum, outperforming traditional methods in sample efficiency and stability.

Approximate Dynamic Programming (ADP) encompasses a suite of methodologies for numerically approximating solutions to sequential decision problems—typically Markov Decision Processes (MDPs)—when their scale, structure, or complexity preclude the use of classical dynamic programming. ADP replaces intractable exact cost-to-go functions or policies with parametrized surrogates, leveraging approaches such as function approximation, optimization over reduced spaces, robustification, or bounding techniques. The objective is to compute near-optimal policies that are efficient to evaluate and implement, with rigorous guarantees on their sub-optimality, sample efficiency, and computational tractability. ADP has found applications in reinforcement learning, control, operations research, and beyond.

1. Mathematical Formulations and Optimization Structures

A key methodological pivot in ADP is the reformulation of the exact dynamic programming solution—typically based on BeLLMan equations or value iteration—as an optimization problem over restricted classes of value functions or policies. For example, in DRADP (“Distributionally Robust Approximate Dynamic Programming”) (Petrik, 2012), the approach optimizes over a set of approximate value functions $\mathcal{V} = \{ v \in V : v = \Phi_v x \}$ and a robustified superset of state-action occupancy measures $(\pi)$, which are themselves parametrized by feature matrices $\Phi_u$. The lower bound on expected return for policy $\pi$ is given by: $$\tilde{\rho}(\pi) = \max_{v \in \mathcal{V}} \left[ \alpha^T v - \frac{1}{1-\gamma} \max_{u \in (\pi)} u^T(A v - b) \right]$$ where $A$ captures BeLLMan residual relationships and $b$ is linked to the reward structure. This formulation enables the policy computation problem to be posed as either a bilinear or mixed-integer linear program (MILP), scalable with respect to feature and sample counts rather than underlying state-action cardinality.

Alternative ADP variants build on the notion of surrogate string optimization (Liu et al., 2014, Liu et al., 2018), casting sequential control problems as monotone string-submodular maximization so that greedy policies represent ADP solutions. In this paradigm, curvatures of the surrogate value-to-go function $g$ determine uniform performance factors.

2. Robustness, Error Bounds, and Theoretical Guarantees

A haLLMark of advanced ADP methods is the derivation of robustness and explicit sub-optimality bounds. DRADP, for instance, ensures that the robust lower bound $\tilde{\rho}(\pi)$ always underestimates or matches the true return, and is tight under mild invertibility and determinism assumptions (Theorem 1 in (Petrik, 2012)): $\rho(\pi) = \tilde{\rho}(\pi)$ Moreover, strong $L_1$-weighted error bounds are attained: $$\|v^* - v_{\bar{\pi}}\|_{1,\alpha} \leq \frac{2}{1-\gamma} \min_{v\in \mathcal{V}} \|v - T v\|_\infty$$ which dominate $L_\infty$-based bounds of traditional approximate linear programming (ALP) and policy iteration (API), leading to tighter, less conservative guarantees even for non-asymptotic (finite iteration) solutions. Under certain smoothness or concentration conditions, these bounds can be further improved using weighted norms.

In the string optimization framework, ADP is shown to achieve at least a fraction $\beta$ of optimal performance, where

$$\beta = \frac{\sum_{i=0}^{K-1}(1-\epsilon_i)}{\sum_{i=0}^{K-1} \eta_i}$$

with forward and elemental curvatures $\epsilon_i$, $\eta_i$ precisely quantifying how closely the problem structure approximates submodularity (Liu et al., 2014, Liu et al., 2018).

3. Practical Scalability, Algorithms, and Computational Aspects

ADP techniques are expressly constructed for scalability. By approximating value functions and occupancy frequencies through low-dimensional basis (feature) representations and by utilizing samples rather than enumerative sweeps, the computational costs are substantially decoupled from the size of the underlying MDP state and action spaces. When formulating DRADP as a bilinear program or MILP, the main computational burden transitions to the number of features and constraints induced by sampled trajectories and problem structure.

Mathematical programming formulations admit the use of fast solvers, and when necessary, further reductions (e.g., via McCormick inequalities to convexify bilinear programs) are available. This enables offline, batch-oriented policy computation—particularly suitable for industrial, robotic, or operations research applications—distilling high-quality policies that can be rapidly evaluated online.

Empirically, ADP implementations have demonstrated the capacity to yield stable, low-variance, and sample-efficient learning, even in small-sample regimes, as illustrated by the DRADP outperforming LSPI and related methods in inverted pendulum and chain MDP benchmarks (Petrik, 2012).

4. Empirical Results and Benchmark Comparisons

In DRADP’s empirical evaluation (Petrik, 2012), policies were learned for the inverted pendulum (using only 9 radial basis plus a constant) and for chain MDPs with 30 states. DRADP consistently maintained high performance and low variance across random instances, achieving maximal step counts in the pendulum despite limited training data. In chain problems, DRADP’s robust policy return exceeded that of API variants in over 1000 trial instances, confirming that theoretical improvements in error bounding translate to practical policy quality.

A point of contrast is the sensitivity of alternative ADP methods (e.g., LSPI or ALP) to sample size and approximation error, while DRADP’s “pessimistic” bounding mitigates instability and produces reproducible improvements under uncertainty and partial information settings.

5. Applicability to Large-Scale MDPs and Uncertainty Handling

States of practical significance frequently arise in domains where MDPs are extremely high-dimensional, often with partially observed or uncertain dynamics. DRADP’s robustification—ensuring the occupancy measure is taken from a superset of feasible distributions—enhances both resilience to model mis-specification and graceful degradation in low-data regimes. Such properties are essential for batch/offline reinforcement learning, where all relevant samples are acquired beforehand, and in industrial control (e.g., robotics, manufacturing scheduling) where exact enumeration is infeasible.

Additionally, ADP’s mathematical programming basis, along with flexible choice of surrogate function classes and occupancy approximators, enables adaptation to uncertainties intrinsic to the system or data. This is highly relevant for safety-critical domains and in reinforcement learning applications under distribution shift or partial observability.

6. Implications for Reinforcement Learning and Future Directions

ADP methods—with strong theoretical foundations and empirical validation—constitute a viable foundation for offline and batch reinforcement learning, offering guaranteed sub-optimality in policy synthesis and resilience to approximation errors. The structural flexibility in choosing representable subspaces, constraint sampling, and robustification paves the way for further integration with modern function approximation (e.g., neural architectures), extension to non-tabular and continuous spaces, and hybridization with model-based or model-free learning.

In summary, ADP, as instantiated in robust formulations such as DRADP (Petrik, 2012), provides a suite of scalable, mathematically principled tools for large-scale sequential decision-making problems, achieving improved convergence, tighter error bounds, and enhanced robustness compared to traditional approximate dynamic programming approaches.