Bellman Flow Constraint

• Bellman flow constraint is a fixed-point condition that aligns measures and value functions with underlying Markovian dynamics in both control and reinforcement learning.
• In continuous-time LQR, it ensures that gradient flows remain within the stabilizing region, guaranteeing convergence to an optimal feedback gain.
• Within diffusion-based successor measure models, it blends one-step and bootstrapped denoising losses to maintain temporal consistency and accurate policy evaluation.

The Bellman flow constraint is a fundamental condition arising in both classical control and modern reinforcement learning (RL) to ensure the consistency of measures, value functions, or generative models with the underlying Markovian dynamics. It appears as a stationarity or fixed-point requirement on policy-evaluation-like quantities, most notably in the context of successor measures and in policy parameterization schemes that align continuous optimization with the Bellman equation. The constraint ensures that, when optimizing over policies or their associated objects, the resulting estimator or controller remains dynamically stable or consistent under repeated application of environment dynamics and policy.

1. Bellman Flow Constraint in Continuous-Time LQR

In the continuous-time infinite-horizon Linear Quadratic Regulator (LQR), the Bellman flow constraint emerges in connection with the Hamilton-Jacobi-Bellman (HJB) equation for policy optimality. If the value function is postulated as $V(x) = x^\top P x$, the HJB equation becomes: $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$ For linear feedback $\mu(x) = -Kx$, stability requires $A_K := A - BK$ to be Hurwitz. The associated matrix Riccati equation, obtained by inserting the value function ansatz and feedback policy into the HJB equation, admits a solution $P_K$ if and only if $K$ stabilizes the system.

The continuous-time Bellman error is defined as the negative trace-residual of the Riccati equation: $$\mathcal{E}(K) := -\mathrm{tr}\left(A^\top P_K + P_K A - P_K B R^{-1} B^\top P_K + Q\right)$$ where $P_K$ is the unique solution to

$A_K^\top P_K + P_K A_K + Q + K^\top R K = 0$

The Bellman flow constraint, in this context, is that the gradient flow induced by $-\nabla_K \mathcal{E}(K)$ preserves stabilizability for all $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$0: $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$1 and along this flow, $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$2 is nonincreasing, and $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$3 remains in the stabilizing region $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$4 so that $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$5 is always Hurwitz. This crucially prevents destabilization during iterative optimization and guarantees convergence to the unique optimal gain $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$6 (Gießler et al., 11 Jun 2025).

2. Bellman Flow Constraint in Successor State Measure Models

For a Markov Decision Process (MDP) $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$7 and a fixed policy $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$8, the successor state measure (SSM) $$0 = x^\top Q x + \mu(x)^\top R \mu(x) + (\nabla_x V(x))^\top (A x + B \mu(x))$$9 assigns to each target state $\mu(x) = -Kx$0 the discounted probability that, starting from $\mu(x) = -Kx$1, the agent reaches $\mu(x) = -Kx$2 under transition kernel $\mu(x) = -Kx$3 and discount $\mu(x) = -Kx$4: $\mu(x) = -Kx$5 The SSM satisfies the Bellman flow (or successor-measure) equation: $\mu(x) = -Kx$6 The Bellman flow constraint here enforces that any parameterized model of the SSM (including neural or diffusion models) must satisfy this fixed-point recursion. This ensures alignment with the true discounted visitation measure under the policy and dynamics (Schramm et al., 2024).

3. Parameterized Representations and the Role of the Constraint

When the successor state measure $\mu(x) = -Kx$7 is represented by a $\mu(x) = -Kx$8-step diffusion model, the Bellman flow constraint leads to a structured, Bellman-like update on the diffusion step distribution. The distribution over entire noisy trajectories is modeled as $\mu(x) = -Kx$9 in the forward direction (with a fixed diffusion/noising kernel) and approximated by a learned reverse chain $A_K := A - BK$0 with neural parameterization of the per-step denoising process.

The imposition of the Bellman flow constraint in this diffusion context translates into requiring that the learned denoising distribution obey a mixture balance: $A_K := A - BK$1 To enforce this, the loss blends a ground-truth denoising step from the one-step transition and a bootstrap denoising step with a target network: $A_K := A - BK$2 This two-term structure derives directly from the Bellman flow constraint and mimics the TD update in classical RL, controlling the temporal evolution of the model so that it converges to the consistent solution of the Bellman equation. This maintains dynamical and probabilistic correctness in the learned SSM (Schramm et al., 2024).

4. Analytic Properties, Domain, and Invariance

In the continuous-time LQR instance, the Bellman flow constraint (that trajectories remain in the stabilizing set) arises analytically because $A_K := A - BK$3 is smooth and coercive on its effective domain:

• The domain is $A_K := A - BK$4, and the essential stabilizing set is $A_K := A - BK$5 is Hurwitz$A_K := A - BK$6.
• The Bellman error is real-analytic on this domain.
• As $A_K := A - BK$7 approaches the boundary of stabilizability or becomes unbounded, $A_K := A - BK$8 and thus $A_K := A - BK$9, ensuring that the level sets are compact and fully contained within the stabilizing region.
• The Bellman flow constraint enforces that gradient flow trajectories never exit the stabilizing region $P_K$0, which is essential for iteration safety in continuous-time policy search and improvement (Gießler et al., 11 Jun 2025).

5. Connections to RL Policy Evaluation, Policy Iteration, and Diffusion Models

The Bellman flow constraint is closely related to classical RL concepts:

• In the LQR setting, the gradient flow interleaves policy evaluation (solving for $P_K$1) and policy improvement (updating $P_K$2 along $P_K$3), creating a continuous-time analogue of Kleinman’s policy iteration.
• In successor measure modeling, enforcement of the constraint imports temporal-difference style variance reduction, trading off a small bias for practical acceleration and stability—mirroring benefits known from TD learning over Monte Carlo methods.
• For diffusion-based SSMs, the Bellman flow constraint is incorporated by blending supervised (one-step) and bootstrapped (multi-step, target network) denoising terms, consistent with the theoretical fixed point of the Bellman equation. This constraint provably upper-bounds the Bellman-flow KL divergence and drives convergence to a true successor measure (Schramm et al., 2024).

6. Practical Implementation and Empirical Observations

For diffusion SSM models, the Bellman flow constraint is implemented by augmenting the per-step loss with both "ground-truth" and "bootstrap" denoising objectives based on sampled transitions and a frozen target network. The on-policy implementation proceeds by:

1. Sampling from the buffer.
2. Drawing a diffusion step and noise.
3. Forming the noisy input.
4. Applying the appropriate loss term depending on whether the sampled state is the immediate successor or requires bootstrapping via the target network.
5. Performing a gradient step on model parameters.

Empirically, the constraint leads to improved accuracy at modeling successor distributions and better downstream policy performance compared to unconstrained diffusion baselines in standard offline RL benchmarks. Using a target network improves stability by decoupling rapid changes in the main network, and the TD-style loss reduces variance per diffusion step. A limitation is that the procedure is only strictly correct on-policy and may become biased with off-policy data (Schramm et al., 2024).

7. Comparative Insights and Theoretical Guarantees

In continuous-time LQR policy search, Bellman-error gradient flow converges linearly to the unique optimal stabilizing feedback. The Bellman flow constraint ensures all iterates remain stabilizing, and empirically, the Bellman-error flow converges faster (in terms of residual norm) than the classical cost-gradient flow, with all trajectories approaching the unique fixed point $P_K$4 (Gießler et al., 11 Jun 2025).

A crucial theoretical property is the invariance of stabilizability under the Bellman flow constraint; sublevel sets of the Bellman error remain strictly within the stabilizing region, forbidding loss of stability during continuous policy optimization. This differentiates methods based on the Bellman flow constraint from those lacking explicit safeguards and underpins robust convergence guarantees in both control and learning settings.

---

References:

• "Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error" (Gießler et al., 11 Jun 2025)
• "Bellman Diffusion Models" (Schramm et al., 2024)