---
title: 'Linear MDPs: A Feature-Based Approach'
url: https://www.emergentmind.com/topics/linear-markov-decision-process-mdp
type: topic
---

# Linear MDPs: A Feature-Based Approach

A linear Markov decision process (linear MDP) is a feature-based reinforcement-learning model in which the reward function and transition kernel admit low-dimensional linear parameterizations with respect to a known feature map. In the episodic formulation, this replaces direct dependence on the cardinality of the state space by dependence on a feature dimension \(d\), making the model a standard theoretical abstraction for large-scale or even infinite-state RL. The framework supports Bellman recursions that remain realizable in the chosen feature class, and it underlies regret, sample-complexity, and structural results ranging from least-squares value iteration to horizon-free learning and low-switching-cost control [2101.00494, 2403.10738].

## 1. Formal model and Bellman structure

A standard \(H\)-horizon episodic linear MDP is an MDP \(\mathcal M=(\mathcal S,\mathcal A,\{P_h,r_h\}_{h=1}^H)\) with a known feature map
\[
\phi:\mathcal S\times\mathcal A\to\mathbb R^d
\]
such that, for each stage \(h\), there are unknown parameters \(\theta_h\in\mathbb R^d\) and \(\mu_h\) for which
\[
r_h(s,a)=\langle \phi(s,a),\theta_h\rangle
\]
and
\[
\Pr[x_{h+1}=x'\mid x_h=s,a_h=a]=\langle \phi(s,a),\mu_h(x')\rangle.
\]
A tabular MDP with \(S\) states and \(A\) actions is the special case \(d=SA\) with \(\phi(s,a)=e_{(s,a)}\) [2101.00494].

A closely related statement uses the same linear structure to express Bellman recursion directly. For any policy \(\pi\),
\[
Q_h^\pi(s,a)=r_h(s,a)+\mathbb E_{s'\sim P_h(\cdot\mid s,a)}[V_{h+1}^\pi(s')],
\qquad
V_h^\pi(s)=\sum_{a\in\mathcal A}\pi_h(a\mid s)Q_h^\pi(s,a),
\]
and the optimal functions satisfy
\[
Q_h^*(s,a)=\phi(s,a)^\top\theta_h+\phi(s,a)^\top\mu_h^\top V_{h+1}^*,
\qquad
V_h^*(s)=\max_{a\in\mathcal A}Q_h^*(s,a).
\]
The associated core assumptions are feature-span or realizability, boundedness of features and parameters, and Bellman completeness, namely that the Bellman operator maps the linear \(Q\)-class into itself [2506.00818].

The time-homogeneous variant keeps a single transition kernel \(P\) and reward distribution \(R\) across the horizon and assumes a known feature map \(\phi:\mathcal S\times\mathcal A\to\mathbb R^d\) with \(\|\phi(s,a)\|_2\le 1\). In that setting, there exist unknown \(\theta_r\in\mathbb R^d\) and \(\mu\in\mathbb R^{|\mathcal S|\times d}\) such that
\[
r(s,a)=\mathbb E[R(s,a)]=\langle \phi(s,a),\theta_r\rangle,
\qquad
P(s'|s,a)=[\mu\,\phi(s,a)]_{s'}.
\]
The same paper imposes the condition that for every \(v\in\mathbb R^{|\mathcal S|}\) with \(\|v\|_\infty\le 1\), one has \(\|\mu^\top v\|_2\le \sqrt d\) [2403.10738].

Under a dataset of \(N\) trajectories whose coverage matrix
\[
\Lambda=\sum_{i=1}^N\sum_{h=1}^H\phi(s_h^i,a_h^i)\phi(s_h^i,a_h^i)^\top
\]
satisfies \(\lambda_{\min}(\Lambda)\ge \rho>0\), one obtains a learned policy \(\hat\pi\) with suboptimality
\[
V_1^*(s_1)-V_1^{\hat\pi}(s_1)=\widetilde O\Bigl(\frac{dH}{\sqrt N}\Bigr),
\]
or, in online RL, regret \(\widetilde O(dH\sqrt T)\). This identifies the central scaling variable as the feature dimension rather than the raw state-space size [2506.00818].

## 2. Least-squares value iteration and low-switching-cost learning

A principal algorithmic instantiation is least-squares value iteration with upper confidence bounds. In the low-switching-cost formulation, the learner performs backward estimation at each episode \(k\) and stage \(h\) using
\[
\Lambda_h^k=\lambda I+\sum_{\tau=1}^{k-1}\phi(x_h^\tau,a_h^\tau)\phi(x_h^\tau,a_h^\tau)^\top,
\]
\[
w_h^k=(\Lambda_h^k)^{-1}\sum_{\tau=1}^{k-1}\phi(x_h^\tau,a_h^\tau)\bigl[r_h(x_h^\tau,a_h^\tau)+\max_a\widetilde Q_{h+1}^k(x_{h+1}^\tau,a)\bigr],
\]
and optimistic estimates
\[
\widetilde Q_h^k(s,a)=\min\Bigl\{(\phi(s,a))^\top w_h^k+\beta\,\|\phi(s,a)\|_{(\Lambda_h^k)^{-1}},\,H\Bigr\}.
\]
The distinctive device is a policy-change test: the learner commits to a new policy only when the learned covariance has doubled in some direction, expressed as
\[
(\Lambda_h^{\tilde k})^{-1}\not\preccurlyeq 2\,(\Lambda_h^k)^{-1}.
\]
Planning then uses the stale optimistic action-value estimate \(Q_h^k\leftarrow \widetilde Q_h^{\tilde k}\) for the entire episode [2101.00494].

With \(\lambda=1\) and
\[
\beta=c\,d\,H\sqrt{\log\bigl(2dKH/\delta\bigr)},
\]
the method achieves, with probability \(1-\delta\),
\[
\mathrm{Regret}(K)=O\bigl(\sqrt{d^3H^4K\log^2(dKH/\delta)}\bigr),
\]
together with global switching cost
\[
N_{\rm switch}=O(dH\log K).
\]
The same paper states that this regret matches the best existing polynomial algorithm in the linear-MDP setting, while the switching cost is exponentially smaller than that comparator’s episode-by-episode updating rule [2101.00494].

The tabular specialization \(d=SA\) yields regret \(\widetilde O(\sqrt{S^3A^3H^4K})\) and local switching cost \(O(SAH\log K)\). The same work also proves that substantial improvement in switching complexity is impossible in general: any deterministic-policy algorithm that guarantees sublinear regret over linear MDPs of dimension \(d\) and horizon \(H\) must incur
\[
N_{\rm switch}^{\rm gl}=\Omega\Bigl(\frac{dH}{\log d}\Bigr).
\]
This lower bound places the \(O(dH\log K)\) guarantee near the optimal switching regime up to logarithmic factors [2101.00494].

The analysis combines least-squares confidence sets, an elliptical-potential lemma of the form
\[
\log\det\Bigl(\sum_{t=1}^T\phi_t\phi_t^\top+\lambda I\Bigr)=O(d\log T),
\]
and a delayed-update argument showing that when updates are postponed, the bonus under the stale policy remains within a factor \(\sqrt 2\) of the latest bonus. The result is a regret decomposition in which infrequent switching does not change the overall statistical order [2101.00494].

## 3. Horizon-free regret and structural lemmas

A distinct line of work studies whether the regret of linear-MDP learning can avoid polynomial dependence on the planning horizon \(H\). In the horizon-free formulation, the learner operates over \(K\) episodes of length \(H\) in a time-homogeneous linear MDP with possibly large or infinite \(|\mathcal S|\), and the total-bounded rewards assumption requires that in any episode
\[
\sum_{h=1}^H r_h \le 1
\quad\text{almost surely}.
\]
Rather than explicitly estimating the transition model and constructing a full stack of inhomogeneous value functions, the method directly estimates value functions and confidence sets [2403.10738].

The algorithm maintains an \(\varepsilon\)-net \(\mathcal W_\varepsilon\) of possible value functions. For each \(v\in\mathcal W_\varepsilon\), a weighted least-squares subroutine estimates \(\mu^\top v\), \(\mu^\top(v^2)\), and an empirical variance. These estimates define a confidence region
\[
\mathcal U^k=\left\{\tilde\mu \,\middle|\, \bigl|\phi^\top\tilde\mu^\top v-\phi^\top\hat\theta^k(v)\bigr|\le b^k(v,\phi)\;\; \forall \phi\in\Phi,\forall v\in\mathcal W_\varepsilon\right\},
\]
with bonus
\[
b^k(v,\phi)=\alpha\sqrt{\phi^\top[\Lambda^k(v)]^{-1}\phi}+4\varepsilon.
\]
A separate reward confidence set \(\Theta^k\) is built using the VOFUL procedure, and optimistic planning solves
\[
(\mu^k,\theta^k)=\arg\max_{\tilde\mu\in\mathcal U^k,\tilde\theta\in\Theta^k}\max_\pi \mathbb E_{\pi,\tilde\mu,\tilde\theta}\Bigl[\sum_{h=1}^H r_h\Bigr].
\]
The resulting regret bound is
\[
\mathrm{Regret}(K)=\widetilde O(d^{5.5}\sqrt K+d^{6.5}),
\]
with no polynomial dependence on \(H\) [2403.10738].

The key structural input is a bound on the total variation of the inhomogeneous optimal value functions. Defining
\[
l_h=\|V_h^*-V_{h+1}^*\|_\infty,
\]
the paper proves
\[
\sum_{h=1}^{H-1} l_h \le 2d,
\qquad
l_h\le \frac{2d}{H-h+1}.
\]
This shows that the cumulative drop of \(V_h^*\) across the horizon is controlled by the feature dimension \(d\) rather than by \(H\). The proof then groups layers into \(O(\log H)\) segments, combines a variance-ratio argument with elliptical-potential control, and removes explicit horizon dependence from the leading regret term [2403.10738].

The same result also identifies current limitations. As stated, the algorithm is not computationally efficient because it enumerates an \(\varepsilon\)-net of the value-function set of size approximately \(\varepsilon^{-d}\). The dependence \(d^{5.5}\sqrt K\) remains far from the lower bound \(\Omega(d\sqrt K)\), and extending horizon-free guarantees to other function-approximation models, including neural networks, is left open [2403.10738].

## 4. Extensions of the linear-MDP framework

One extension replaces the linear reward model by a generalized linear model while retaining linear transitions. A generalized linear MDP (GLMDP) is specified by reward features \(\phi_r:\mathcal S\times\mathcal A\to\mathbb R^{d_r}\), transition features \(\phi_p:\mathcal S\times\mathcal A\to\mathbb R^{d_p}\), a known link function \(g\), and unknown parameters \(\theta_h,\mu_h\) satisfying
\[
g\!\bigl(\mathbb E[r_h\mid s,a]\bigr)=\phi_r(s,a)^\top\theta_h,
\qquad
P_h(s'\mid s,a)=\phi_p(s,a)^\top\mu_h(s').
\]
The associated function class is
\[
\mathcal F_h=\left\{(s,a)\mapsto g(\phi_r(s,a)^\top\theta)+\phi_p(s,a)^\top\beta:\theta\in\mathbb R^{d_r},\,\beta\in\mathbb R^{d_p}\right\},
\]
and, under mild regularity, the Bellman operator maps \(\mathcal F_{h+1}\) into \(\mathcal F_h\), so \(Q_h^*\in\mathcal F_h\) [2506.00818].

The offline algorithms in that setting are Generalized Pessimistic Value Iteration (GPEVI) and its semi-supervised variant SS-GPEVI. GPEVI constructs reward and transition confidence sets, then forms pessimistic estimates
\[
\hat Q_h(s,a)=\min_{\theta\in\mathcal C_{r,h},\,\beta\in\mathcal C_{p,h}}
\Bigl\{g^{-1}(\phi_r^\top\theta)+\phi_p^\top\beta\Bigr\}.
\]
With probability at least \(1-\delta\), GPEVI satisfies
\[
V_1^*(s)-V_1^{\hat\pi}(s)
=\widetilde O\!\Bigl(\sqrt{\tfrac{d_rH^2}{n}}+\sqrt{\tfrac{(d_p+d_r)^2H^4}{n}}\Bigr),
\]
while SS-GPEVI with \(n\) labeled and \(N\) unlabeled samples yields
\[
V_1^*(s)-V_1^{\hat\pi_{\rm SS}}(s)
=\widetilde O\!\Bigl(\sqrt{\tfrac{d_rH^2}{n}}+\sqrt{\tfrac{(d_p+d_r)^2H^4}{n+N}}\Bigr).
\]
Setting \(g^{-1}(x)=x\) and \(\phi_r=\phi_p\) recovers the standard linear MDP as a special case [2506.00818].

Another extension is performative RL with linear MDPs, in which deploying a policy changes both rewards and transitions. In the infinite-horizon discounted setting, the policy-dependent environment is modeled by
\[
r_\pi(s,a)=\langle \phi(s,a),\theta_\pi\rangle,
\qquad
P_\pi(s'\mid s,a)=\langle \phi(s,a),\mu_\pi(s')\rangle.
\]
The associated discounted occupancy measure
\[
d^\pi(s,a)=\mathbb E\Bigl[\sum_{t=0}^\infty \gamma^t\,\mathbf 1\{s_t=s,a_t=a\}\Bigr]
\]
leads to a regularized primal problem over occupancies and feature moments. Repeated regularized retraining produces a two-step recurrence
\[
\|d_{t+1}-d_S\|\le \beta_1\|d_t-d_S\|+\beta_2\|d_{t-1}-d_S\|,
\]
with \(\beta_{1,2}=O((\varepsilon_\theta+\gamma\sqrt D\,\varepsilon_\mu)/(\lambda\sqrt\kappa))\). For sufficiently large \(\lambda\), the iterates converge linearly to a performatively stable solution, and in the finite-sample regime empirical repeated retraining converges under a bounded-coverage condition. The same framework is applied to stochastic Stackelberg and multi-follower coarse correlated equilibrium settings [2411.05234].

A further hybridization couples known linear time-invariant plant dynamics
\[
x_{t+1}=Ax_t+Bu_t
\]
with an exogenous feature-based linear Markov process
\[
P(s_{t+1}\mid s_t)=\phi(s_t)^T\mu_t(\cdot).
\]
In that model, the optimal finite-horizon state-action value function admits an explicit parametric form involving a Riccati recursion for \(G_t\), and the optimal policy is closed form:
\[
\pi_t^*(x,s)=K_{t,x}x+K_{t,s}s+K_{t,h}\sum_{i=1}^d \phi_i(s)\,\overline h_{i,t+1}.
\]
Least-Squares Value Iteration then estimates only the unknown feature-based coefficients, without explicit transition estimation or exploration bonuses because the environment is exogenous. The paper provides input-to-state stability guarantees and a regret bound whose leading term scales as \(\widetilde O(T\sqrt{dL})\) [2508.17185].

## 5. Related but distinct linear structures in MDP theory

The term “linear MDP” should be distinguished from “linearly-solvable MDP” despite the common acronym LMDP. In a linearly-solvable MDP, the model is an undiscounted first-exit control problem with passive dynamics \(P(s'|s)\), state reward \(R(s)\le 0\), and a control cost given by a Kullback–Leibler divergence:
\[
\mathcal R(s,a)=R(s)-\lambda\,\mathrm{KL}[a(\cdot|s)\|P(\cdot|s)].
\]
Introducing the desirability function
\[
Z(s)=\exp(V(s)/\lambda)
\]
turns the Bellman equation into the linear relation
\[
Z(s)=\exp(R(s)/\lambda)\sum_{s'}P(s'|s)Z(s'),
\]
with optimal policy
\[
a^*(s'|s)=\frac{P(s'|s)Z(s')}{\sum_y P(y|s)Z(y)}.
\]
The same linearity supports Z-learning, hierarchical decomposition, and task compositionality. Empirically, the hierarchical method reported faster convergence than \(\varepsilon\)-greedy Q-learning in the Taxi domain and higher throughput in the AGV domain [1603.03267].

A second distinct line uses linear programming to solve generic MDPs approximately. For a discounted finite MDP with Bellman operator \(T\), the exact LP is
\[
\min c^\top J \quad\text{subject to}\quad J\ge TJ.
\]
Approximate linear programming replaces \(J\) by \(\Phi r\), giving
\[
\min c^\top \Phi r \quad\text{subject to}\quad \Phi r\ge T(\Phi r),
\]
while the generalized reduced linear program (GRLP) replaces the full set of \(nd\) constraints by nonnegative linear combinations:
\[
\min c^\top\Phi r \quad\text{subject to}\quad W^\top\Phi r\ge W^\top T\Phi r,
\]
where \(W\in\mathbb R_+^{(nd)\times m}\) has full column rank. The analysis introduces two \(\|\cdot\|_\infty\)-contraction operators, \(\Gamma\) and \(\tilde\Gamma\), and derives the error bound
\[
\|J^*-\hat J\|_{1,c}\le \frac{6\|J^*-\Phi r^*\|_\infty+2\delta}{1-\alpha},
\qquad
\delta=\|\Gamma J^*-\tilde\Gamma J^*\|_\infty.
\]
This is a theory of linear approximation and constraint reduction for solving large MDPs, not the feature-based linear-MDP model for transition and reward generation [1409.3536].

## 6. Limitations, misconceptions, and open questions

A common source of confusion is terminological. In the standard linear-MDP model, rewards and transitions are linear in features, but the Bellman operator itself is not thereby made linear in the value function. By contrast, linearly-solvable MDPs achieve a genuinely linear Bellman equation after the exponential change of variables \(Z(s)=\exp(V(s)/\lambda)\) [1603.03267]. Another common misconception is that linear MDPs are inherently finite-state; the horizon-free analysis explicitly allows \(|\mathcal S|\) to be large or infinite [2403.10738].

Several limitations are explicit in the current theory. Horizon-free regret is presently known through an algorithm that is not computationally efficient because it enumerates an \(\varepsilon\)-net of the value-function class, and its \(\widetilde O(d^{5.5}\sqrt K+d^{6.5})\) rate is still far from the \(\Omega(d\sqrt K)\) lower bound [2403.10738]. Low-switching-cost learning is nearly optimal in its switching dependence, but the lower bound \(\Omega(dH/\log d)\) shows that no-regret deterministic-policy algorithms must still pay a nontrivial number of policy changes [2101.00494].

The generalizations also sharpen the boundary of the model. GLMDPs broaden the reward class to generalized linear models, but their stated sample-efficiency guarantee matches standard linear-MDP rates only up to an extra factor of \(H\) in the horizon dependence [2506.00818]. Performative RL with linear MDPs requires explicit sensitivity, feature-regularity, and bounded-coverage assumptions, and the finite-sample guarantees depend polynomially on the feature dimension and target accuracy through quantities such as \(D^5\), \((1-\gamma)^{-2}\), and \(\delta^{-4}\) [2411.05234].

Taken together, these results position the linear MDP as a central abstraction for feature-based RL: expressive enough to model large-scale problems, structured enough to admit Bellman completeness and sharp statistical guarantees, and flexible enough to support extensions to generalized rewards, performative environments, and hybrid control systems. The main unresolved issues are computational efficiency under the strongest guarantees, tighter dependence on \(d\) and \(H\), and the transfer of these guarantees beyond linear feature classes [2403.10738, 2506.00818].

Source: https://www.emergentmind.com/topics/linear-markov-decision-process-mdp