Linear MDPs: A Feature-Based Approach

• Linear MDPs are feature-based models where rewards and transitions admit low-dimensional linear parameterizations, enabling scalability beyond finite state spaces.
• They underpin Bellman recursions that remain realizable within the feature class, with algorithms like least-squares value iteration yielding near-optimal regret and low switching costs.
• Extensions to generalized rewards, horizon-free regret methods, and performative RL enhance the framework's flexibility while highlighting ongoing computational challenges.

A linear Markov decision process (linear MDP) is a feature-based reinforcement-learning model in which the reward function and transition kernel admit low-dimensional linear parameterizations with respect to a known feature map. In the episodic formulation, this replaces direct dependence on the cardinality of the state space by dependence on a feature dimension $d$, making the model a standard theoretical abstraction for large-scale or even infinite-state RL. The framework supports Bellman recursions that remain realizable in the chosen feature class, and it underlies regret, sample-complexity, and structural results ranging from least-squares value iteration to horizon-free learning and low-switching-cost control (Gao et al., 2021, Zhang et al., 2024).

1. Formal model and Bellman structure

A standard $H$-horizon episodic linear MDP is an MDP $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$ with a known feature map

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$

such that, for each stage $h$, there are unknown parameters $\theta_h\in\mathbb R^d$ and $\mu_h$ for which

$r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$

and

$$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$

A tabular MDP with $S$ states and $H$0 actions is the special case $H$1 with $H$2 (Gao et al., 2021).

A closely related statement uses the same linear structure to express Bellman recursion directly. For any policy $H$3,

$H$4

and the optimal functions satisfy

$H$5

The associated core assumptions are feature-span or realizability, boundedness of features and parameters, and Bellman completeness, namely that the Bellman operator maps the linear $H$6-class into itself (Zhang et al., 1 Jun 2025).

The time-homogeneous variant keeps a single transition kernel $H$7 and reward distribution $H$8 across the horizon and assumes a known feature map $H$9 with $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$0. In that setting, there exist unknown $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$1 and $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$2 such that

$$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$3

The same paper imposes the condition that for every $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$4 with $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$5, one has $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$6 (Zhang et al., 2024).

Under a dataset of $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$7 trajectories whose coverage matrix

$$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$8

satisfies $$\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)$$9, one obtains a learned policy $\phi:\mathcal S\times\mathcal A\to\mathbb R^d$0 with suboptimality

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$1

or, in online RL, regret $\phi:\mathcal S\times\mathcal A\to\mathbb R^d$2. This identifies the central scaling variable as the feature dimension rather than the raw state-space size (Zhang et al., 1 Jun 2025).

2. Least-squares value iteration and low-switching-cost learning

A principal algorithmic instantiation is least-squares value iteration with upper confidence bounds. In the low-switching-cost formulation, the learner performs backward estimation at each episode $\phi:\mathcal S\times\mathcal A\to\mathbb R^d$3 and stage $\phi:\mathcal S\times\mathcal A\to\mathbb R^d$4 using

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$5

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$6

and optimistic estimates

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$7

The distinctive device is a policy-change test: the learner commits to a new policy only when the learned covariance has doubled in some direction, expressed as

$\phi:\mathcal S\times\mathcal A\to\mathbb R^d$8

Planning then uses the stale optimistic action-value estimate $\phi:\mathcal S\times\mathcal A\to\mathbb R^d$9 for the entire episode (Gao et al., 2021).

With $h$0 and

$h$1

the method achieves, with probability $h$2,

$h$3

together with global switching cost

$h$4

The same paper states that this regret matches the best existing polynomial algorithm in the linear-MDP setting, while the switching cost is exponentially smaller than that comparator’s episode-by-episode updating rule (Gao et al., 2021).

The tabular specialization $h$5 yields regret $h$6 and local switching cost $h$7. The same work also proves that substantial improvement in switching complexity is impossible in general: any deterministic-policy algorithm that guarantees sublinear regret over linear MDPs of dimension $h$8 and horizon $h$9 must incur

$\theta_h\in\mathbb R^d$0

This lower bound places the $\theta_h\in\mathbb R^d$1 guarantee near the optimal switching regime up to logarithmic factors (Gao et al., 2021).

The analysis combines least-squares confidence sets, an elliptical-potential lemma of the form

$\theta_h\in\mathbb R^d$2

and a delayed-update argument showing that when updates are postponed, the bonus under the stale policy remains within a factor $\theta_h\in\mathbb R^d$3 of the latest bonus. The result is a regret decomposition in which infrequent switching does not change the overall statistical order (Gao et al., 2021).

3. Horizon-free regret and structural lemmas

A distinct line of work studies whether the regret of linear-MDP learning can avoid polynomial dependence on the planning horizon $\theta_h\in\mathbb R^d$4. In the horizon-free formulation, the learner operates over $\theta_h\in\mathbb R^d$5 episodes of length $\theta_h\in\mathbb R^d$6 in a time-homogeneous linear MDP with possibly large or infinite $\theta_h\in\mathbb R^d$7, and the total-bounded rewards assumption requires that in any episode

$\theta_h\in\mathbb R^d$8

Rather than explicitly estimating the transition model and constructing a full stack of inhomogeneous value functions, the method directly estimates value functions and confidence sets (Zhang et al., 2024).

The algorithm maintains an $\theta_h\in\mathbb R^d$9-net $\mu_h$0 of possible value functions. For each $\mu_h$1, a weighted least-squares subroutine estimates $\mu_h$2, $\mu_h$3, and an empirical variance. These estimates define a confidence region

$\mu_h$4

with bonus

$\mu_h$5

A separate reward confidence set $\mu_h$6 is built using the VOFUL procedure, and optimistic planning solves

$\mu_h$7

The resulting regret bound is

$\mu_h$8

with no polynomial dependence on $\mu_h$9 (Zhang et al., 2024).

The key structural input is a bound on the total variation of the inhomogeneous optimal value functions. Defining

$r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$0

the paper proves

$r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$1

This shows that the cumulative drop of $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$2 across the horizon is controlled by the feature dimension $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$3 rather than by $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$4. The proof then groups layers into $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$5 segments, combines a variance-ratio argument with elliptical-potential control, and removes explicit horizon dependence from the leading regret term (Zhang et al., 2024).

The same result also identifies current limitations. As stated, the algorithm is not computationally efficient because it enumerates an $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$6-net of the value-function set of size approximately $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$7. The dependence $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$8 remains far from the lower bound $r_h(s,a)=\langle \phi(s,a),\theta_h\rangle$9, and extending horizon-free guarantees to other function-approximation models, including neural networks, is left open (Zhang et al., 2024).

4. Extensions of the linear-MDP framework

One extension replaces the linear reward model by a generalized linear model while retaining linear transitions. A generalized linear MDP (GLMDP) is specified by reward features $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$0, transition features $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$1, a known link function $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$2, and unknown parameters $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$3 satisfying

$$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$4

The associated function class is

$$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$5

and, under mild regularity, the Bellman operator maps $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$6 into $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$7, so $$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$8 (Zhang et al., 1 Jun 2025).

The offline algorithms in that setting are Generalized Pessimistic Value Iteration (GPEVI) and its semi-supervised variant SS-GPEVI. GPEVI constructs reward and transition confidence sets, then forms pessimistic estimates

$$\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.$$9

With probability at least $S$0, GPEVI satisfies

$S$1

while SS-GPEVI with $S$2 labeled and $S$3 unlabeled samples yields

$S$4

Setting $S$5 and $S$6 recovers the standard linear MDP as a special case (Zhang et al., 1 Jun 2025).

Another extension is performative RL with linear MDPs, in which deploying a policy changes both rewards and transitions. In the infinite-horizon discounted setting, the policy-dependent environment is modeled by

$S$7

The associated discounted occupancy measure

$S$8

leads to a regularized primal problem over occupancies and feature moments. Repeated regularized retraining produces a two-step recurrence

$S$9

with $H$00. For sufficiently large $H$01, the iterates converge linearly to a performatively stable solution, and in the finite-sample regime empirical repeated retraining converges under a bounded-coverage condition. The same framework is applied to stochastic Stackelberg and multi-follower coarse correlated equilibrium settings (Mandal et al., 2024).

A further hybridization couples known linear time-invariant plant dynamics

$H$02

with an exogenous feature-based linear Markov process

$H$03

In that model, the optimal finite-horizon state-action value function admits an explicit parametric form involving a Riccati recursion for $H$04, and the optimal policy is closed form: $H$05 Least-Squares Value Iteration then estimates only the unknown feature-based coefficients, without explicit transition estimation or exploration bonuses because the environment is exogenous. The paper provides input-to-state stability guarantees and a regret bound whose leading term scales as $H$06 (Makdah et al., 24 Aug 2025).

5. Related but distinct linear structures in MDP theory

The term “linear MDP” should be distinguished from “linearly-solvable MDP” despite the common acronym LMDP. In a linearly-solvable MDP, the model is an undiscounted first-exit control problem with passive dynamics $H$07, state reward $H$08, and a control cost given by a Kullback–Leibler divergence: $H$09 Introducing the desirability function

$H$10

turns the Bellman equation into the linear relation

$H$11

with optimal policy

$H$12

The same linearity supports Z-learning, hierarchical decomposition, and task compositionality. Empirically, the hierarchical method reported faster convergence than $H$13-greedy Q-learning in the Taxi domain and higher throughput in the AGV domain (Jonsson et al., 2016).

A second distinct line uses linear programming to solve generic MDPs approximately. For a discounted finite MDP with Bellman operator $H$14, the exact LP is

$H$15

Approximate linear programming replaces $H$16 by $H$17, giving

$H$18

while the generalized reduced linear program (GRLP) replaces the full set of $H$19 constraints by nonnegative linear combinations: $H$20 where $H$21 has full column rank. The analysis introduces two $H$22-contraction operators, $H$23 and $H$24, and derives the error bound

$H$25

This is a theory of linear approximation and constraint reduction for solving large MDPs, not the feature-based linear-MDP model for transition and reward generation (Lakshminarayanan et al., 2014).

6. Limitations, misconceptions, and open questions

A common source of confusion is terminological. In the standard linear-MDP model, rewards and transitions are linear in features, but the Bellman operator itself is not thereby made linear in the value function. By contrast, linearly-solvable MDPs achieve a genuinely linear Bellman equation after the exponential change of variables $H$26 (Jonsson et al., 2016). Another common misconception is that linear MDPs are inherently finite-state; the horizon-free analysis explicitly allows $H$27 to be large or infinite (Zhang et al., 2024).

Several limitations are explicit in the current theory. Horizon-free regret is presently known through an algorithm that is not computationally efficient because it enumerates an $H$28-net of the value-function class, and its $H$29 rate is still far from the $H$30 lower bound (Zhang et al., 2024). Low-switching-cost learning is nearly optimal in its switching dependence, but the lower bound $H$31 shows that no-regret deterministic-policy algorithms must still pay a nontrivial number of policy changes (Gao et al., 2021).

The generalizations also sharpen the boundary of the model. GLMDPs broaden the reward class to generalized linear models, but their stated sample-efficiency guarantee matches standard linear-MDP rates only up to an extra factor of $H$32 in the horizon dependence (Zhang et al., 1 Jun 2025). Performative RL with linear MDPs requires explicit sensitivity, feature-regularity, and bounded-coverage assumptions, and the finite-sample guarantees depend polynomially on the feature dimension and target accuracy through quantities such as $H$33, $H$34, and $H$35 (Mandal et al., 2024).

Taken together, these results position the linear MDP as a central abstraction for feature-based RL: expressive enough to model large-scale problems, structured enough to admit Bellman completeness and sharp statistical guarantees, and flexible enough to support extensions to generalized rewards, performative environments, and hybrid control systems. The main unresolved issues are computational efficiency under the strongest guarantees, tighter dependence on $H$36 and $H$37, and the transfer of these guarantees beyond linear feature classes (Zhang et al., 2024, Zhang et al., 1 Jun 2025).