Piecewise Linear ADP

• Piecewise linear ADP is a family of methods that approximates value functions using segmented linear or affine components, preserving Bellman structure and enabling efficient region-wise operations.
• These approaches leverage structured partitioning, symbolic regression, and mixed-integer optimization to overcome dimensionality challenges in continuous and hybrid MDPs.
• They provide explicit error bounds, convergence guarantees, and practical benefits in applications such as Mars-rover control and adaptive cruise control.

Searching arXiv for relevant papers on piecewise linear-based approximate dynamic programming. {"query":"piecewise linear approximate dynamic programming arXiv piecewise linear value function MDP", "max_results": 10} Piecewise linear-based approximate dynamic programming denotes a family of ADP methods in which the value function, the $Q$-function, or associated policy surrogates are represented by piecewise-defined linear or affine components, and Bellman updates are carried out either exactly within that representation or approximately through projection, compression, or regression. Across the literature, the term covers several distinct but related constructions: rectangular piecewise-linear convex value functions for continuous-state MDPs (Feng et al., 2012), XADD-based symbolic value functions with bounded-error compression for hybrid MDPs (Vianna et al., 2013), $(\min,+)$ linear approximations whose lower envelopes are piecewise-linear (Lakshminarayanan et al., 2014, Lakshminarayanan et al., 2014), and max-affine or difference-of-max formulations used for constrained piecewise affine control and mixed-integer encodings (He et al., 2023, Teichrib et al., 2024). The unifying objective is to replace unstructured discretization or generic nonlinear approximators by representations that preserve Bellman structure, admit efficient region-wise operations, and, in several cases, support explicit error, stability, or optimality guarantees.

1. Problem classes and motivation

Piecewise linear-based ADP appears in settings where exact dynamic programming is impeded by the dimensionality or heterogeneity of the state space, yet the problem data already have significant geometric structure. In the continuous-state MDP setting of Feng et al., the state space is $X=[0,1)^d$, the action set is finite, each action has a finite set of stochastic successor shifts, and the finite-horizon Bellman recursion is

$$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$

The central observation is that structured reward and transition models induce large regions on which the value function is constant or piecewise-linear, so exhaustive gridding is unnecessarily expensive (Feng et al., 2012).

The same motivation reappears in hybrid MDPs, where states combine Boolean and continuous variables and exact symbolic dynamic programming can represent value functions exactly but may generate extremely large decision diagrams. Vianna et al. characterize the resulting growth as often exponential in the number of regions, which motivates bounded-error compression rather than exact preservation of every partition boundary (Vianna et al., 2013).

In infinite-horizon discounted MDPs, the $(\min,+)$ line of work treats approximation as projection onto a subsemimodule instead of projection onto a conventional linear subspace. Here the approximation is still piecewise-linear, but the geometry is governed by lower envelopes of basis functions under idempotent algebra. This alters both the projected Bellman equation and the error analysis, with emphasis on $L_\infty$ rather than least-squares criteria (Lakshminarayanan et al., 2014, Lakshminarayanan et al., 2014).

For constrained piecewise affine systems, the motivation is partly computational and partly certifiable. Hybrid MPC can solve such problems, but its offline and online burden may be large. The PWA-ADP formulation of Kim et al. uses piecewise affine value surrogates and penalty functions so that online control reduces either to a smaller-scale MILP or to a single convex quadratic program, while preserving stability and safety statements under quantified approximation error (He et al., 2023).

A further motivation arises when a value function has already been sampled offline. Teichrib and Schulze Darup formulate piecewise regression as a mixed-integer optimization problem, aiming at a globally optimal fit within a chosen piecewise model class rather than a gradient-trained neural approximation. This suggests a complementary use of piecewise linear ADP: exact or approximate dynamic programming can generate training targets, and mixed-integer piecewise regression can compress them into a control-oriented surrogate (Teichrib et al., 2024).

2. Representation families

The literature does not use a single canonical representation. Instead, several piecewise-linear function classes are employed, each aligned with a different Bellman operator and computational substrate.

Framework	Value-function form	Core structure
RPWLC	$V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$ on rectangle $D_j$	Rectangular partition with local $\alpha$-vectors
XADD	Case statement with affine leaves	DAG of Boolean and linear tests
$(\min,+)$ ADP	$(\min,+)$0	Lower envelope over a subsemimodule
PWA / max-affine regression	$(\min,+)$1 or $(\min,+)$2	Polyhedral regions induced by active pieces

In the structured continuous-state MDP framework, the value function is represented over an axis-aligned rectangular partition of $(\min,+)$3 into hyper-rectangles $(\min,+)$4, and on each region $(\min,+)$5 the function is convex piecewise-linear:

$(\min,+)$6

The resulting representation is termed rectangular piecewise-linear convex, or RPWLC (Feng et al., 2012).

In hybrid symbolic DP, the XADD stores a case-defined piecewise-linear function as a DAG whose internal nodes test Boolean variables or linear inequalities and whose leaves are affine functions of the continuous state. Each leaf can be interpreted as valid on a union of convex polytopes specified by the path conditions. This representation is more general in partition geometry than rectangular decompositions, since regions need not be axis-aligned (Vianna et al., 2013).

In the $(\min,+)$7 framework, approximation takes the form

$(\min,+)$8

Because the approximate value is the lower envelope of affine or affine-like basis functions, it is piecewise-linear and concave when the $(\min,+)$9 are affine in $X=[0,1)^d$0. The basis family determines the breakpoints and the shape of the approximation (Lakshminarayanan et al., 2014).

The 2014 companion treatment formulates the same idea as a $X=[0,1)^d$1 subsemimodule $X=[0,1)^d$2 with

$X=[0,1)^d$3

again yielding a piecewise-linear approximation, but now emphasized as a fixed-point iteration for either state values or state-action values (Lakshminarayanan et al., 2014).

Control-oriented PWA ADP and MIP-based regression expand the representational range. Kim et al. use a convex max-affine surrogate

$X=[0,1)^d$4

with regions induced by active affine pieces (He et al., 2023). Teichrib and Schulze Darup use the more general form

$X=[0,1)^d$5

which becomes any PWA function when $X=[0,1)^d$6, since it is a difference of two convex PWA functions (Teichrib et al., 2024). A common misconception is that piecewise linear ADP is inherently limited to convex value functions; these two-max constructions show that non-convex continuous piecewise surfaces are also directly representable.

3. Bellman updates, projection operators, and region-wise recursion

In RPWLC dynamic programming, Bellman backup proceeds region by region. For an action $X=[0,1)^d$7 and a model region $X=[0,1)^d$8, one forms the action-value surface

$X=[0,1)^d$9

Because both $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$0 and the shifted restrictions of $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$1 are PWLC, $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$2 is also PWLC. The successor sum is implemented through repeated cross-sums of local $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$3-vector sets, and the action maximization is implemented by union of $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$4-sets followed by pruning. In the notation of the paper,

$$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$5

with LP-based dominance pruning after each cross-sum or union (Feng et al., 2012).

The exactness of this backup depends on partition refinement. Regions are subdivided when different outcomes require different supports or when shifted copies of a region intersect the current value partition in different ways. The backup therefore alternates between symbolic Bellman arithmetic and geometric partition management. This suggests that approximation is not always introduced through the function form itself; in some formulations, the piecewise-linear machinery supports exact DP over a structured continuous model.

In XADD-based symbolic DP, the Bellman regression for a hybrid MDP with continuous action parameters $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$6 is

$$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$7

followed by

$$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$8

Addition, multiplication, Dirac-based integration, discrete summation, and continuous maximization are all carried out symbolically within the XADD formalism (Vianna et al., 2013).

The $$V^{k+1}(x)=\max_{a\in A}\left\{R_a(x)+\sum_{(\delta x_i,p_i)\in \Delta^a} p_i\,V^k(x+\delta x_i)\right\}.$$9 literature replaces region-wise Bellman arithmetic by projected fixed-point equations. In the infinite-horizon discounted setting,

$(\min,+)$0

and the approximate solution is defined by the $(\min,+)$1 Projected Bellman Equation

$(\min,+)$2

The projection $(\min,+)$3 selects the pointwise smallest element of the subsemimodule that dominates the Bellman image, so the approximation is controlled by an order-theoretic projection rather than orthogonality or least-squares residual minimization (Lakshminarayanan et al., 2014).

The companion formulation makes the iteration explicit:

$(\min,+)$4

where $(\min,+)$5 is the $(\min,+)$6 projection and $(\min,+)$7 is the $(\min,+)$8-Bellman operator. The closed form

$(\min,+)$9

shows that the projection step is analytically simple once $L_\infty$0 is known (Lakshminarayanan et al., 2014).

For constrained PWA systems, the Bellman operator is modified by a PWA penalty:

$L_\infty$1

The penalty $L_\infty$2 is itself constructed as a min-max of affine violations over a union-of-polyhedra constraint set, so the stage cost remains PWA and the one-step update stays compatible with mixed-integer encoding (He et al., 2023).

4. Data structures, optimization routines, and computational mechanisms

The algorithmic content of piecewise linear-based ADP lies in how Bellman updates are executed without expanding every region or candidate hyperplane. In the RPWLC framework, partitions are stored as kd-trees whose internal nodes split on one coordinate and whose leaves store a region plus its current $L_\infty$3-set. Cross-sum and union operations walk two kd-trees in tandem, splitting only when necessary. After backup, adjacent hyperrectangles with identical PWLC vector sets are merged by a single depth-first pass of the kd-tree, preserving kd-tree consistency (Feng et al., 2012).

Dominance pruning is critical in this setting. Every time a cross-sum or union generates a candidate set of linear facets, a small LP is solved for each candidate vector to determine whether it is dominated on the region: “Is $L_\infty$4?” If so, the vector is discarded. This incremental pruning, adapted from POMDP methods, is the main reason the local number of surviving facets often remains small even when the raw cross-sum would yield $L_\infty$5 candidates (Feng et al., 2012).

XADD compression addresses a different combinatorial bottleneck. Two affine leaves are merged by replacing them with a single affine approximation valid on the union of their path conditions, with the new coefficients chosen to minimize the worst-case pointwise error over those regions. The original saddle-point problem is recast as a bilevel LP, and solved by constraint generation: worst-case vertices are generated by region-wise LPs, added to a master LP, and the process terminates when no new constraint increases the objective. The bounded-error compression operator XADDComp then performs bottom-up pairwise leaf merging subject to a user-specified tolerance $L_\infty$6 (Vianna et al., 2013).

In the $L_\infty$7 projection approach, the computational bottleneck is shifted from combinatorial partition management to repeated Bellman evaluation and coefficient updates. The MPADP algorithm solves a finite-dimensional program equivalent to the projected Bellman equation by coordinate-wise updates

$L_\infty$8

The projection subproblem itself has closed form, and the paper also notes that the $L_\infty$9-dimensional problem decouples into scalar subproblems under the associated simplification (Lakshminarayanan et al., 2014). The 2014 companion paper further introduces a variational or weak projection using test functions $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$0, reducing projection to a smaller LP when $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$1 (Lakshminarayanan et al., 2014).

Mixed-integer optimization enters in two different roles. For regression, the difference-of-max approximation

$V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$2

is trained by an MIQP with big-$V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$3 linearization, auxiliary variables $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$4, and binary selectors $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$5 for active segments at each sample. Symmetry breaking and pre-clustering are proposed to reduce binary count, and when $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$6 after clustering the formulation may reduce to a single QP (Teichrib et al., 2024).

For constrained PWA control, online policy evaluation is encoded as an MILP whose binaries select both the active mode of the PWA dynamics and the active affine piece of the successor value surrogate. An alternative explicit policy is learned offline as a PWA actor and then projected onto the admissible input set by a convex QP, yielding an online map that is itself PWA (He et al., 2023). A plausible implication is that piecewise linear ADP spans a spectrum from exact symbolic recursion to offline statistical fitting, with mixed-integer programming serving either as a Bellman subroutine or as a surrogate-learning engine.

5. Approximation guarantees, convergence, and certification

Several strands of the literature provide formal guarantees, but the guarantees differ substantially in type.

The XADD-compression approach provides explicit sup-norm error accumulation bounds. If each compression at stage $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$7 incurs at most $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$8 error in sup norm, then the cumulative error after $V(x)=\max_{\ell\in L_j}\{w_\ell^\top x+b_\ell\}$9 undiscounted stages is at most $D_j$0, and in the discounted case

$D_j$1

This is significant because approximation is inserted inside symbolic DP itself rather than after value iteration, yet the effect remains explicitly bounded (Vianna et al., 2013).

In the $D_j$2 projected Bellman framework, the main statement is an approximation bound relative to the best representable value in the chosen subsemimodule. If

$D_j$3

then the MPPBE solution satisfies

$D_j$4

The proof uses contraction of the Bellman operator in $D_j$5, a shifted dominating approximation, and monotonicity of the projection (Lakshminarayanan et al., 2014).

The related $D_j$6 ADP scheme in finite-state MDPs establishes that the composite operator $D_j$7 is a contraction with factor $D_j$8, so the iteration converges to the unique fixed point $D_j$9, and similarly for the $\alpha$0-value iteration. Its error bound is

$\alpha$1

where $\alpha$2 is the best $\alpha$3 fit in the chosen basis class (Lakshminarayanan et al., 2014).

The regression-based piecewise approximation of Teichrib and Schulze Darup gives a different guarantee. Theorem 1 states that there exists $\alpha$4 such that the MIQP is equivalent to the original nonlinear least-squares problem, so any optimizer is globally optimal in the training loss and no local optima exist. The same work also gives an implicit regularization statement in the PWA case, where the big-$\alpha$5 constraints impose a priori bounds on the row norms of the affine coefficients, and a continuity result showing that the fitted function can be embedded as an MI-feasible block inside a larger control or DP-based optimization (Teichrib et al., 2024).

For constrained PWA systems, the guarantees extend beyond approximation quality to closed-loop behavior. Under stabilizability assumptions and sufficiently large penalty weight, each soft value-iteration iterate is continuous and PWA, and the sequence converges pointwise to a soft-constrained value limit. If the approximation error satisfies

$\alpha$6

then the implicit policy yields asymptotic stability within a sublevel set. The suboptimality bound for the implicit policy is

$\alpha$7

and an analogous bound is given for the explicit projected policy under a small enough policy error. The same paper develops offline MILP tests for finite-horizon descent, positive invariance, and reachability, thereby turning Lyapunov-like conditions into verifiable optimization problems (He et al., 2023).

A common misconception is that piecewise linear ADP necessarily sacrifices formal guarantees once approximation is introduced. The literature instead shows multiple guarantee regimes: contraction-based convergence, sup-norm error propagation bounds, global optimality of fitted surrogate parameters within a model class, and stability or safety certificates for closed-loop deployment.

6. Empirical performance, applications, and comparative perspective

The empirical record shows that the benefits of piecewise linear structure are highly problem-dependent. In the Mars-rover benchmarks of Feng et al., three solvers were compared: naive uniform-grid DP, structured RPWC DP, and structured RPWLC DP. In 1D, the overhead of kd-trees and pruning outweighed the savings. In 2D, at fine resolutions of at least 200 grid points, the structured methods were orders of magnitude faster than naive gridding, with log$\alpha$8-time gains greater than 4–6. In 3D, the naive solver timed out beyond roughly $\alpha$9 grid cells, whereas the structured piecewise-linear method handled more than $(\min,+)$0 in minutes. The paper also reports that up to $(\min,+)$1 of the continuous state remained one big rectangle with a single linear-facet description (Feng et al., 2012).

The bounded symbolic DP results exhibit a similar trade-off in a more general hybrid setting. On MarsRover1D, exact SDP at iteration 6 produced an XADD with about $(\min,+)$2 nodes, while $(\min,+)$3 reduced node count by $(\min,+)$4 with sup-norm error below $(\min,+)$5. On MarsRover2D, exact SDP yielded about $(\min,+)$6 nodes by iteration 6; $(\min,+)$7 reduced this to about $(\min,+)$8, produced a $(\min,+)$9 wall-clock speedup, and kept worst-case pointwise error below $(\min,+)$00. On InventoryControl with 10 continuous resources, $(\min,+)$01 kept node count near $(\min,+)$02 and time per iteration under $(\min,+)$03 s (Vianna et al., 2013).

The $(\min,+)$04 methods were evaluated on both discrete and continuous domains. In Grid-World $(\min,+)$05 with $(\min,+)$06, the reported value-function error was about $(\min,+)$07, compared with an ALP bound of about $(\min,+)$08; with $(\min,+)$09, the reported error was about $(\min,+)$10 versus $(\min,+)$11. In Mountain-Car, with basis centers on a $(\min,+)$12 grid and $(\min,+)$13, the greedy policy reached the goal in about 150 steps and was described as near-optimal (Lakshminarayanan et al., 2014). The companion paper emphasizes that such performance is linked to the suitability of the $(\min,+)$14 envelope for capturing upper-bound geometry in $(\min,+)$15 (Lakshminarayanan et al., 2014).

For constrained control, the case studies involve an inverted pendulum with elastic walls and an adaptive cruise control problem. In the pendulum example, after 10 value-iteration steps the explicit policy achieved fast regulation, the verification framework identified invariant and reachable sets, and online computation was reported as $(\min,+)$16 for the explicit policy versus about $(\min,+)$17 s for the implicit MILP policy and about $(\min,+)$18 s for hybrid MPC. In adaptive cruise control, over 100 scenarios, hybrid MPC with $(\min,+)$19 averaged about $(\min,+)$20 s per step with cost $(\min,+)$21 and $(\min,+)$22 safety; the implicit policy averaged about $(\min,+)$23 s with cost $(\min,+)$24 and $(\min,+)$25 safety; the explicit policy averaged about $(\min,+)$26 s with cost $(\min,+)$27 and $(\min,+)$28 safety (He et al., 2023).

The MIP-based regression results address approximation quality rather than Bellman iteration directly. The authors report that even for $(\min,+)$29 or $(\min,+)$30, the fitted surfaces can capture highly nonlinear value functions with sub-$(\min,+)$31 MSE on test grids while keeping binary count small enough for modern MIP solvers (Teichrib et al., 2024). This suggests a practical division of labor: dynamic programming generates data, and globally optimized piecewise regression compresses that data into a deployment-ready surrogate.

From a comparative standpoint, piecewise linear-based ADP should not be identified with a single algorithmic doctrine. Some methods are exact but structure-exploiting; some are approximate with bounded Bellman-stage error; some rely on idempotent-algebraic projection; some are aimed at safety-critical constrained control; and some are primarily surrogate-fitting tools for downstream use. What binds them is not a common implementation, but a common commitment to represent Bellman-relevant functions by geometrically interpretable linear pieces, so that approximation, optimization, and certification can be carried out in a form compatible with the problem’s combinatorial and polyhedral structure.