Commit to the Bit: Reactive Reinforcement Learning Done Right

Abstract: Reinforcement learning algorithms are commonly analyzed (and designed) under the Markov assumption. This is unrealistic, as most environments encountered in practice are either partially observable, or require function approximation that restricts the agent to access non-Markovian state features. We consider the problem of learning an optimal reactive policy in a finite environment with deterministic observations (or equivalently, hard state aggregation). We introduce a new algorithm, Committed Q-learning, and prove almost-sure convergence to the optimal reactive policy under an intuitive assumption we call rewire-robustness. This assumption is strictly weaker than the $q_\star$-realizability condition used in prior work. Our algorithm is a variant of classical Q-learning in which the behavior policy commits to a single action upon entering a feature, and only resamples actions when the observed feature changes. A crucial part of our analysis is the introduction of quasi-Markov environments.

• The paper introduces Committed Q-learning, leveraging committed exploration to overcome state aggregation challenges in reactive reinforcement learning.
• It formalizes quasi-Markov environments and introduces rewire-robustness as a key structural criterion for converging to optimal policies.
• Empirical results in corridor environments demonstrate that Committed Q-learning rapidly outperforms classical Q-learning under partial observability.

Committed Q-learning: Foundations and Advances in Reactive RL

Introduction

The classical theoretical treatment of reinforcement learning (RL) is contingent on the Markov assumption: the agent observes sufficient information at each timestep to ensure future transitions and rewards depend only on the current state and action. However, practical deployments frequently violate this assumption, either due to inherent partial observability or the use of function approximation that constrains the received features. Most prior theoretical analyses require value function “realizability,” demanding that the optimal value be constant within each aggregated state or feature. This condition is often unnecessarily restrictive.

The paper "Commit to the Bit: Reactive Reinforcement Learning Done Right" (2605.28276) revisits value-based RL under hard state aggregation (deterministic observations). It introduces Committed Q-learning, a modification of classical Q-learning designed for reactive policies, which provably learns optimal policies under weaker assumptions than prior work. Central to the analysis is the formalization of quasi-Markov environments and the introduction of "rewire-robustness," an intuitive and verifiable structural criterion for successful policy recovery.

Problem Setting and Hard State Aggregation

Consider RL in an episodic POMDP with a deterministic feature mapping $\varphi: \mathcal{X} \to \mathcal{Z}$, producing a finite set of “features.” A reactive policy maps features to action distributions (“options”). The principal challenge is that Markovian assumptions no longer hold—states aggregated into the same feature may have divergent dynamics or reward structures, impeding the applicability of Bellman-style value iteration and Q-learning.

Classically, convergence results require $q_\star$-realizability: the optimal $Q$-values must be constant within each feature. This is rare in practical settings, as illustrated by the simple corridor environment, where policy optimality depends on entrance distributions, not the average or stationary values inside the feature.

Committed Q-learning Algorithm

Committed Q-learning modifies the exploration protocol: an option is sampled only upon entering a new feature, and the agent “commits” to that option until a feature change. This slight behavioral constraint aligns the effective learning process with the (coarser) feature-level dynamics, enabling tractable analysis even when intra-feature state information is not Markov.

The critical change is the synchronous update and persistent option commitment during residence in a feature, thus avoiding spurious value mixing from state-conditional transitions obscured by aggregation.

Quasi-Markov Environments and Bellman Risk

A key technical contribution is the characterization of quasi-Markov environments: systems where each feature $z$ possesses a unique entrance distribution over underlying states. In such cases, the Markov property is “approximately” retained at the level of feature transitions, making dynamic programming viable across the aggregate MDP.

Standard objectives like value error or Bellman error are shown to be ill-suited for learning in these settings, as they can promote non-optimal reactive policies. Instead, the notion of Bellman risk is advanced, where the relevant value assigned to a feature is the (distribution-weighted) value upon entrance. The value function minimizing Bellman risk suffices for optimal greedy policy selection in quasi-Markov environments.

Rewire-Robustness: Optimality Under Minimal Assumptions

While quasi-Markovness is a sufficient condition, Committed Q-learning operates correctly even in environments where this does not strictly hold, provided “rewire-robustness” is satisfied. Rewire-robustness formalizes the requirement that modifying the entrance mechanics of a feature (without altering intra-feature dynamics or overall transitions between features) does not change the set of optimal reactive policies.

This is strictly weaker than $q_\star$-realizability—enabling optimal learning in more complex, practical settings. The main theoretical result proves almost sure convergence of Committed Q-learning to the optimal reactive policy under the assumption of rewire-robustness.

Theoretical Guarantees

The main theorem establishes that provided all options are sufficiently explored, the environment is proper (all policies terminate), and stepsizes are scheduled appropriately, the iterates of Committed Q-learning converge almost surely to a fixed-point solution $Q_\star$. The greedy policy with respect to $Q_\star$ is optimal in any rewire-robust environment; it coincides with the optimal reactive policy in the underlying MDP when $q_\star$-realizability holds, and even more generally when only rewire-robustness is satisfied.

Crucially, the proof leverages recent advances in stochastic approximation under Markovian noise [liu2025ode], circumventing limitations of classical stochastic approximation theorems. The effective dynamics learned by Committed Q-learning can be interpreted as operating on a “$\pi$-MDP”: the aggregate MDP induced by the behavioral policy's stationary entrance distributions.

Empirical Results

Empirical evaluation in the corridor environment starkly demonstrates the limitations of classical (non-committed) Q-learning under state aggregation: it fails even in trivial environments. In contrast, Committed Q-learning rapidly and reliably achieves optimal policy performance.

(Figure 1)

Figure 1: Learning curves of Committed Q-learning and regular Q-learning in corridor environments of varying lengths; only Committed Q-learning achieves rapid and stable convergence to optimality.

Structural Implications and Related Literature

From a foundational perspective, this work shifts the lens of RL theory from global value function realizability to the finer structure of how feature-level transitions interact with policy class restrictions. The insights on Bellman risk, aggregate MDPs, and Bayesian disaggregation matrices connect to classic dynamic programming with state aggregation [bertsekas2012dynamic], but identify more precise criteria and effective algorithms for general RL settings.

The analysis also clarifies the relationship among multiple notions: $q_\star$-realizability $q_\star$0 (generalized) rewire-robustness $q_\star$1 quasi-Markovness, with the latter being a particularly useful structural bridge.

Practical and Theoretical Implications

From an algorithmic standpoint, Committed Q-learning's modifications are straightforward to implement and do not require additional memory of past features—only commitment to options until a feature transition. The results also have implications for feature engineering in deep RL: selecting or learning state abstractions that preserve rewire-robustness guarantees tractability of value-based methods for reactive policies, even in large or partially observable environments.

Theoretically, these concepts inform the design of function approximation schemes, the interpretation of minimum Bellman risk solutions, and lay the groundwork for structured RL in non-Markovian or partially observable scenarios.

Discussion and Future Directions

The analysis exposes the limits of reactive policies and the necessary conditions for their sufficiency. It also highlights inherent complexity barriers: optimal policy selection under state aggregation remains NP-hard in general [littman1994memoryless], but becomes tractable under rewire-robustness.

Several future directions arise:

• Extending Committed Q-learning analysis and guarantees to non-stationary or temporally extended options;
• Adapting the theory to infinite-horizon discounted or average-reward settings;
• Quantifying robustness when the rewire-robustness assumption is only approximately satisfied;
• Investigating connections to function approximation architectures for learned abstractions in modern deep RL.

Conclusion

Committed Q-learning is characterized by its effectiveness in environments where classical Q-learning fails due to partial observability or restrictive feature aggregation. By establishing convergence and optimality under the minimal rewire-robustness assumption, it advances both the theoretical landscape and practical applicability of RL in real-world, non-Markovian settings. The interplay between aggregate MDP theory, Bellman risk, and algorithm design elucidated here opens promising avenues for both principled RL research and practical reinforcement learning under partial observability.