Bellman Residual Minimization (BRM)

• Bellman Residual Minimization (BRM) is a framework that minimizes the discrepancy between an approximation and its Bellman update, ensuring consistency in value or policy estimation.
• It is applied in various settings including linear value-function approximation, Q-learning, and policy-based forms, each addressing unique geometric and sampling challenges.
• BRM offers performance guarantees and error bounds, but its nonconvex and nonsmooth landscapes often necessitate double-sampling or adversarial reformulations for robust implementation.

Bellman Residual Minimization (BRM) denotes a family of methods that estimate value functions, action-value functions, or policies by directly minimizing the discrepancy between an approximate solution and its Bellman update. In linear value-function approximation, this appears as

$g^*=\arg\min_g \|\Phi g-T(\Phi g)\|_\Xi,$

while in empirical Q-learning it is commonly instantiated as

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$

and in policy-based form as

$\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$

Across these formulations, BRM is a direct Bellman-consistency criterion rather than a direct return-maximization criterion, and the literature repeatedly returns to the tension between those two viewpoints (Ciosek, 2013, Yin et al., 2022, Geist et al., 2016).

1. Core formulation and problem classes

The Bellman residual is the deviation between both sides of the Bellman equation under an approximation. For a policy-evaluation problem with linear architecture, BRM minimizes the unprojected Bellman error in a weighted norm, whereas for Q-learning with function approximation it minimizes the squared discrepancy between $Q_\theta(s,a)$ and a bootstrapped target $r+\gamma \max_{a'}Q_\theta(s',a')$ (Ciosek, 2013, Yin et al., 2022). In policy-based BRM, the criterion is transferred from value functions to policies themselves by optimizing $\|T_*v_\pi-v_\pi\|_{1,\nu}$, where $T_*$ is the Bellman optimality operator and $\nu$ is the sampling distribution (Geist et al., 2016).

Control-oriented BRM extends the same idea to optimality equations. In that setting, the control Bellman operator is

$$(TQ_\theta)(s,a)=R(s,a)+\gamma \sum_{s'}P(s'|s,a)\max_{a'}Q_\theta(s',a'),$$

and the control Bellman residual objective is

$f(\theta)=\frac{1}{2}\|TQ_\theta-Q_\theta\|_2^2.$

The resulting optimization problem is nonconvex and nonsmooth because of the $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$0, but it remains a direct residual criterion for policy optimization rather than policy evaluation (Lee et al., 26 Jan 2026).

The same residual-minimization principle also appears outside standard discounted RL. In stochastic shortest path problems, the Bellman residual of the dynamic programming operator is used to compute suboptimality bounds for approximate solutions (Hansen, 2012). In Hamilton-Jacobi-Bellman PDE solvers, residual minimization is applied to the HJB PDE associated to the Nonlinear Quadratic Regulator problem, with the residual evaluated at collocation points (Borovykh et al., 2022). This breadth of usage suggests that BRM is best understood as a general Bellman-consistency principle whose precise meaning depends on the operator, the norm, and the approximation class.

2. Geometric interpretation and relation to TD and LSTD

A central classical distinction is between minimizing the Bellman residual and solving a projected Bellman fixed point. In the linear case, BRM seeks

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$1

which leads to a least-squares solution of the form

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$2

Geometrically, BRM is the orthogonal projection of the reward vector onto the space spanned by the temporal-difference features $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$3 (Scherrer, 2010, Ciosek, 2013).

By contrast, TD(0) and LSTD solve projected Bellman equations. LSTD is the fixed point of the projected Bellman operator $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$4, admits an instrumental-variables derivation, and corresponds geometrically to an oblique projection rather than an orthogonal one (Ciosek, 2013). The unified oblique-projection view shows that both TD(0) and BRM can be written as special cases of an oblique projection $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$5, differing only in the direction along which projection is taken (Scherrer, 2010).

This comparison has two recurring consequences. First, BRM enjoys a direct performance guarantee, while TD(0) does not in general. Second, LSTD has favorable statistical properties in the standard single-trajectory setting, whereas BRM generally requires two independent samples of successor states to obtain a consistent estimator because the noise is correlated with the regressor $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$6 (Ciosek, 2013). The comparison is therefore not merely about which quantity is minimized; it is also about projection geometry, identifiability, and sampling regime.

3. Performance guarantees, proxy bounds, and the “bad proxy” problem

Several papers derive explicit error bounds for BRM. In policy evaluation with linear approximation, minimizing the Bellman residual controls value-function error through

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$7

where $$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$8 is a concentration coefficient (Scherrer, 2010). In stochastic shortest path problems, Bellman-residual-based suboptimality bounds can be computed even when not all policies are proper, provided one works from a uniformly improvable value function and, in the positive-transition-cost case, uses an explicit upper bound on expected steps to termination (Hansen, 2012).

For policy-based BRM, the main proxy bound is

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(Q_\theta(s_i,a_i)-r(s_i,a_i)-\gamma \max_{a'\in A}Q_\theta(s'_i,a')\right)^2,$$9

The coefficient $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$0 is a concentrability coefficient that measures mismatch between the optimization distribution $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$1 and the discounted occupancy measure of the optimal policy (Geist et al., 2016). The same paper shows that policy-based Bellman residual minimization is a good proxy only when $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$2 is well matched to $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$3, with the ideal case $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$4 yielding concentrability $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$5. Since $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$6 is unknown, that ideal case is typically unattainable (Geist et al., 2016).

The empirical side of this literature is notably skeptical. On randomly generated Garnet MDPs, direct maximization of the mean value is robust to distribution mismatch, whereas policy-based BRM degrades rapidly as concentrability worsens; low residual does not guarantee low value error in the states of interest (Geist et al., 2016). An analogous phenomenon appears in neural Q-learning: extensive experiments comparing Temporal Difference and Residual Gradient show that TD learns policies with higher accumulated reward and greater robustness to parameter perturbation, even when RG achieves a smaller Bellman residual (Yin et al., 2022). The same work emphasizes a key difference between RL and supervised learning: in RL, a small Bellman residual error can correspond to a bad policy, whereas in supervised learning the test loss function is a standard index to indicate performance (Yin et al., 2022).

A common misconception is therefore that Bellman self-consistency is equivalent to good control. The literature does not support that equivalence. What it supports is a more qualified statement: residual minimization can upper-bound error under specific geometry and sampling assumptions, but those assumptions can be loose, hard to verify, or poorly aligned with the states that matter for decision quality.

4. Optimization landscape, hardness, and algorithmic design

From an optimization standpoint, BRM is attractive because it defines a clear scalar objective, but difficult because that objective is often nonconvex, nonsmooth, or both. For neural function approximation, the Mean Squared Bellman Error is highly non-convex and can be non-coercive. Critical-point analysis shows that when global minima exist and the Jacobian has full column rank, over-parametrized neural networks can avoid suboptimal local minima; this motivates Gauss Newton Residual Gradient algorithms that use complete derivatives of the MSBE rather than semi-gradients (Gottwald et al., 2021). The same analysis reports local quadratic convergence to a global minimum in the exact discrete-state variant and demonstrates feasibility in continuous control with sampled updates (Gottwald et al., 2021).

For control problems, the control Bellman residual objective is continuous, locally Lipschitz, and piecewise quadratic over regions defined by greedy-action patterns. Its Clarke subdifferential can be written explicitly, and generalized gradient descent with Armijo backtracking converges to Clarke-stationary points; in the tabular case, the unique stationary point coincides with $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$7 (Lee et al., 26 Jan 2026). A differentiable “soft” variant replaces $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$8 with log-sum-exp, producing the smooth Soft Control Bellman Residual (SCBR) objective and enabling standard gradient descent (Lee et al., 26 Jan 2026).

A separate line of work seeks global or contraction-aligned formulations. Approximate Bilinear Programming minimizes $\min_{\pi\in P}\|T_*v_\pi-v_\pi\|_{1,\nu}.$9, weighted $Q_\theta(s,a)$0, or hybrid norms of the Bellman residual and provides strong a priori guarantees on both robust and expected policy loss, but solving the corresponding bilinear program optimally is NP-hard because Bellman-residual minimization itself is NP-hard (Petrik et al., 2010). More recently, soft BRM with weighted $Q_\theta(s,a)$1-norms has been proposed to align the optimization objective with the contraction geometry of the Bellman operator as $Q_\theta(s,a)$2 increases. In that framework,

$Q_\theta(s,a)$3

and the effective contraction rate approaches the $Q_\theta(s,a)$4-contraction rate as $Q_\theta(s,a)$5, while remaining compatible with gradient-based optimization (Yang et al., 8 Apr 2026). This suggests a systematic route for reducing the geometric mismatch between $Q_\theta(s,a)$6-style objectives and $Q_\theta(s,a)$7-style Bellman contraction.

5. Sampling, double-sampling, and statistical guarantees

A persistent technical obstacle for BRM is the double-sample problem. When Bellman residual minimization is formulated as minimizing a squared conditional expectation, unbiased stochastic gradients are generally unavailable from a single next-state sample (Swamy et al., 2022). This difficulty is one reason BRM has often been considered less tractable than TD-style methods in model-free settings (Ciosek, 2013, Lee et al., 26 Jan 2026).

One response is a game-theoretic reformulation. Conditional moment matching frames BRM as a Conditional Moment Restriction problem and replaces squared residual minimization by a regularized min-max objective,

$Q_\theta(s,a)$8

For BRM, this yields a saddle-point problem in the value function and an adversarial test function. The resulting game can be solved by a no-regret online learning reduction, avoids the double-sample issue, handles nonlinear function classes, and explicitly accounts for finite sample uncertainty (Swamy et al., 2022).

A second response appears in distributional off-policy evaluation. The Energy Bellman Residual Minimizer (EBRM) replaces supremum-extended discrepancies by expectation-extended statistical distances and uses the energy distance to define an empirical Bellman residual over return distributions. The method comes with a finite-sample error bound under realizability and a multi-step extension whose error bound improves behavior under non-realizability, notably without requiring the completeness assumption (Hong et al., 2024).

Recent statistical analysis has also sharpened the offline-learning picture for BRM. For SGDA applied to a BRM minimax formulation, a single Lyapunov potential that couples runs on neighboring datasets yields an $Q_\theta(s,a)$9 on-average argument-stability bound and a matching $r+\gamma \max_{a'}Q_\theta(s',a')$0 excess risk bound, without variance reduction, extra regularization, or restrictive independence assumptions on minibatch sampling. The results hold for standard neural-network parameterizations and minibatch SGD (Kang et al., 26 Aug 2025). This is a marked change from the older view of BRM as principally an optimization object with weak statistical understanding.

6. Extensions beyond classical value prediction

BRM has increasingly been adapted to domains whose structure differs substantially from standard discounted MDPs. In HJB PDE solving for the Nonlinear Quadratic Regulator, a deep neural approximation $r+\gamma \max_{a'}Q_\theta(s',a')$1 is first warm-started by supervised learning on a gradient-augmented synthetic dataset produced by a state-dependent Riccati equation control law, and only then refined by minimizing the HJB residual. The combination of supervised learning and residual minimization avoids spurious solutions and mitigate the data inefficiency of a supervised learning-only approach (Borovykh et al., 2022).

In combinatorial optimization, TreeDQN reformulates branching in branch-and-bound as a tree MDP with a Bellman operator that aggregates over child nodes and is contracting in mean when $r+\gamma \max_{a'}Q_\theta(s',a')$2. Because tree sizes are long-tailed, the learning objective is changed from ordinary mean-squared error to Mean Squared Logarithmic Error, which makes the network optimize the geometric mean of the expected tree size (Sorokin et al., 2023). In approximate dynamic programming, High Dimensional Model Representation is used to store Bellman functions compactly and to relax the required minimizations into trust-region problems sharing the same matrix, thereby reducing memory demand and accelerating approximate Bellman minimization (Pištěk, 2012).

The most visible recent adaptation is to LLM reasoning. Trajectory Bellman Residual Minimization (TBRM) moves from token-level to trajectory-level Bellman residuals, uses the model’s own logits as $r+\gamma \max_{a'}Q_\theta(s',a')$3-values, removes the need for critics, importance-sampling ratios, or clipping, and operates with only one rollout per prompt (Yuan et al., 21 May 2025). The paper proves convergence to the near-optimal KL-regularized policy from arbitrary off-policy data and reports that, on Qwen2.5-Math-7B, TBRM used 33% less GPU memory and achieved a 1.3x speedup over PPO; with $r+\gamma \max_{a'}Q_\theta(s',a')$4, it matched or slightly outperformed GRPO at similar resource use, while GRPO was 3.1x slower than TBRM. Across six standard math reasoning datasets, TBRM consistently outperforms or matches PPO and GRPO, and improved average benchmark accuracy by 1.3% (absolute) over GRPO (Yuan et al., 21 May 2025).

These extensions do not erase the classical criticisms of BRM as a proxy for return maximization. They do, however, show that residual minimization is no longer a single classical algorithmic template. It has become a broad design space spanning orthogonal and oblique projections, bilinear and saddle-point reformulations, smooth and nonsmooth control objectives, weighted $r+\gamma \max_{a'}Q_\theta(s',a')$5 geometries, and trajectory-level formulations adapted to sparse-reward sequence models.