Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration

Abstract: Dynamic programming is one of the most fundamental methodologies for solving Markov decision problems. Among its many variants, Q-value iteration (Q-VI) is particularly important due to its conceptual simplicity and its classical contraction-based convergence guarantee. Despite the central role of this contraction property, it does not fully reveal the geometric structure of the Q-VI trajectory. In particular, when one is interested not only in the final limit $Q<sup>*$ but also in when the induced greedy policy becomes effectively optimal, the standard contraction argument provides only a coarse characterization. To formalize this notion, we denote by $\mathcal X<sup>*$ the set of $Q$-functions whose corresponding tie-broken greedy policies are optimal, referred to as the practically optimal solution set (POS). In this paper, we revisit discounted Q-VI through the lens of switching system theory and derive new geometric insights into its behavior. In particular, we show that although Q-VI does not reach $Q<sup>*$ in finite time in general, it identifies the optimal action class in finite time. Furthermore, we prove that the distance from the iterate to a particular subset of $\mathcal X<sup>*$ decays exponentially at a rate governed by the joint spectral radius (JSR) of a restricted switching family. This rate can be strictly faster than the standard $γ$ rate when the restricted JSR is strictly smaller than $γ$, while the convergence of the entire $Q$-function to $Q<sup>*$ can still be dominated by the slower $γ$ mode, where $γ$ denotes the discount factor. These results reveal a two-stage geometric behavior of Q-VI: a fast convergence toward $\mathcal X_1$, followed by a slower convergence toward $Q<sup>*$ in general.

• The paper demonstrates that Q-VI identifies optimal policies in finite iterations by entering a tube around the Practically Optimal Solution Set (POSS) before full convergence to Q*.
• It employs a switching system framework and joint spectral radius bounds to separate fast policy identification dynamics from slower value refinement.
• The geometric insights offer practical strategies for early stopping and adaptive adjustments in reinforcement learning algorithms.

Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration (2604.17457)

Introduction and Motivation

The paper rigorously examines the geometry of Q-value iteration (Q-VI) for discounted finite Markov Decision Processes (MDPs), departing from conventional contraction-based analysis centered primarily around the Bellman fixed point $Q^*$. While the contraction property of the Bellman operator ensures exponential convergence to $Q^*$ at rate $\gamma$ (the discount factor), this does not elucidate when and how the greedy policy derived from intermediate $Q$-functions becomes optimal—i.e., when iterates induce practically optimal policies, not just optimal values. The analysis introduces the concept of the Practically Optimal Solution Set (POSS), $\mathcal{X}^*$: the set of $Q$-functions whose induced (tie-broken greedy) policies are optimal, even if the $Q$-functions themselves are suboptimal.

By leveraging switching systems theory, the work establishes a geometric and spectral understanding of Q-VI, revealing that policy identification can occur much faster than the pointwise convergence to $Q^*$. The core technical advance is the connection of Q-VI dynamics to joint spectral radius (JSR) bounds in directions orthogonal to the all-ones vector, enabling sharper, two-phase convergence analyses and quantification of finite-time policy identification.

Switching System Representation and Geometric Structure

The Bellman operator for Q-VI can be reformulated as an affine switching system:

$$Q_{k+1} - Q^* = \gamma P \Pi_{Q_k}(Q_k - Q^*) + \gamma P(\Pi_{Q_k} - \Pi_{Q^*})Q^*,$$

where $P$ encodes transition probabilities, and $Q^*$0 encodes the deterministic (tie-broken) greedy policy selection for the current $Q^*$1. Each iteration thus corresponds to a dynamical system that switches among subsystems based on policy selection derived from $Q^*$2.

This leads to crucial geometric sets:

• $Q^*$3 (Affine Invariant Set): The affine set $Q^*$4, consisting of $Q^*$5 plus uniform shifts. All $Q^*$6 in $Q^*$7 induce the same set of optimal policies, i.e., a shift along $Q^*$8 does not affect policy selection.
• POSS $Q^*$9: The set of all $\gamma$0 whose induced greedy policy is optimal; $\gamma$1.

The pivotal geometric insight is that Q-VI, while asymptotically approaching $\gamma$2, enters a finite tube around $\gamma$3 (contained in $\gamma$4) in finite time. Once inside this tube, the greedy policy is guaranteed to be optimal, and further convergence only refines the value estimate without changing the optimal action selection.

Policy Identification and Spectral Rate Analysis

The standard contraction framework obscures the distinction between directions parallel and orthogonal to $\gamma$5. The present analysis uses orthogonal projection to isolate the non-trivial contraction dynamics:

• The error $\gamma$6 is decomposed into a part parallel to $\gamma$7 and an orthogonal component $\gamma$8.
• The projected error $\gamma$9 evolves according to a restricted linear switched system whose modes are associated with the set of possible greedy (possibly stochastic) policies.

The central technical result is that the distance from $Q$0 to $Q$1 (i.e., the norm of $Q$2) contracts at a rate dictated by the JSR of the restricted family $Q$3, $Q$4, considered on the orthogonal complement of $Q$5. When this JSR $Q$6 is strictly smaller than $Q$7, policy identification happens at a rate faster than the global contraction, and entry into the POSS can be sharply quantified.

This establishes two-stage behavior for Q-VI:

1. Stage 1 (Policy Identification): Rapid, exponential contraction in directions orthogonal to $Q$8—entry into the tube around $Q$9 (and thus the POSS) is governed by the restricted JSR.
2. Stage 2 (Value Refinement): Continued contraction toward $\mathcal{X}^*$0 along the all-ones direction at the standard rate $\mathcal{X}^*$1, with policy remaining optimal.

Once the POSS is entered, the subsequent error dynamics become linear (and, in the case of a unique optimal policy, non-switching), governed specifically by the spectral properties of the optimal policy's transition matrix.

Practical and Theoretical Implications

Stronger Claims Highlighted:

• Q-VI identifies the optimal action class in finite iterations, despite possibly requiring infinite time to converge precisely to $\mathcal{X}^*$2.
• The rate of convergence to the set where the policy is optimal (POSS) can be strictly faster than the classical $\mathcal{X}^*$3 rate if the projected JSR is smaller. This is especially pronounced in MDPs where policy classes rapidly stabilize, but value estimates require further refinement.

Numerical Results emphasize that the tube entrance and policy identification occur well before full $\mathcal{X}^*$4 convergence, and the geometric theory accurately predicts this transient behavior—even in stochastic Q-learning.

Implications for RL Algorithms:

• This analysis provides rigorous guarantees on when a greedy policy can be safely extracted during the iterative process, suggesting early stopping criteria and more efficient practical implementations.
• The JSR-based rate offers a tool for sharp convergence analysis beyond generic contraction bounds, which could inform the design of adaptive or step-size strategies for both value- and policy-based RL methods.

Theoretical Ramifications:

• The separation between policy and value convergence modes clarifies longstanding observations regarding policy stabilization in dynamic programming.
• The connection to switching system stability and JSR places RL convergence guarantees within a broader control-theoretic framework, suggesting further connections with non-linear and hybrid systems.

Speculation on Future AI Developments:

• More refined analysis of RL algorithms may focus increasingly on transient geometric phenomena, not only asymptotics.
• Algorithms could incorporate online estimation of JSR bounds to dynamically adapt phases of learning—a possibility for improved sample efficiency and robustness.
• These geometric techniques could be extended to deep RL and large-scale MDPs, leveraging approximate or low-dimensional JSR estimation.

Conclusion

This work provides an in-depth geometric and spectral analysis of Q-value iteration, revealing that fast and finite policy identification is possible well before $\mathcal{X}^*$5 is reached. The identification and utilization of the restricted JSR as the key rate-determining factor advances the understanding of RL dynamics beyond traditional contraction-based methods. The multi-stage convergence characterization has both theoretical significance for understanding RL, and practical implications for the deployment and analysis of RL algorithms in stochastic control and learning domains.