Committed Q-learning: Models & Methods

• Committed Q-learning is a family of methods that integrate commitment into either the control objective or the behavior process, overcoming the limitations of ordinary greedy Q-learning.
• It encompasses distinct formulations, including quasi-hyperbolic control for precommitted agents and reactive Q-learning for deterministic observations with hard state aggregation.
• The approach employs dual learning tables and specialized fixed-point analyses to ensure convergence and optimality under conditions like rewire-robustness.

Searching arXiv for papers on "Committed Q-learning" and closely related formulations. Committed Q-learning denotes a family of Q-learning formulations in which commitment is built into either the control objective or the behavior process, rather than arising from ordinary greedy action selection alone. In arXiv usage represented by "Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs" (Eshwar, 7 Sep 2025) and "Commit to the Bit: Reactive Reinforcement Learning Done Right" (Eberhard et al., 27 May 2026), the term has two technically distinct meanings. One concerns precommitted control for agents with quasi-hyperbolic time preferences; the other concerns reactive reinforcement learning under deterministic observations or hard state aggregation, where the learner commits to an option until the observed feature changes. The two lines share the idea that ordinary one-table Q-learning is insufficient because the relevant value object is not the standard stationary Markovian action-value function.

1. Scope and principal meanings

The term encompasses at least two non-equivalent constructions. In the quasi-hyperbolic setting, commitment is part of the preference model: a precommitted agent chooses in advance a policy that maximizes a time-inconsistent discounted return. In the reactive partially observable setting, commitment is behavioral: an option is sampled upon entering a feature and then maintained while that feature persists.

Formulation	Commitment mechanism	Learned or optimized object
Precommitted QH control	first-step policy $\mu$, then stationary $\pi$ forever	$Q_*^{\sigma,\gamma}$ for time $0$, $Q_*^\gamma$ for times $t \ge 1$
Reactive committed Q-learning	sample option on feature entry, keep it until feature changes	$Q(z,\omega)$ over feature-option pairs

This distinction matters because the two settings solve different problems. The quasi-hyperbolic formulation is a finite-state, finite-action MDP with time-inconsistent preferences, while the reactive formulation addresses finite episodic control with latent states and deterministic observations. A recurrent misconception is to treat all uses of “committed Q-learning” as simple reward shaping or as a generic conservative variant of Q-learning. The cited formulations instead alter either the policy class or the effective Bellman structure.

2. Precommitted control in quasi-hyperbolic discounted MDPs

In the quasi-hyperbolic formulation, the MDP is

$$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$

with finite $\mathcal S,\mathcal A$, transition kernel $\mathcal P(\cdot\mid s,a)$, reward $\pi$0, and QH parameters $\pi$1. The return is

$\pi$2

equivalently with discount sequence

$\pi$3

Immediate reward is therefore weighted by $\pi$4, while all future rewards are downweighted by the extra present-bias factor $\pi$5. When $\pi$6, the model reduces to ordinary exponential discounting.

The preference model distinguishes naive, sophisticated, and precommitted agents. Naive agents do not recognize future time inconsistency and keep replanning. Sophisticated agents anticipate future selves and seek subgame-perfect equilibria. Precommitted agents understand their time inconsistency but commit in advance to a policy that maximizes total QH-discounted return from the initial time. The committed control objective is

$\pi$7

where $\pi$8 is a one-step non-stationary policy,

$\pi$9

The central structural theorem is that arbitrary non-stationary policies are unnecessary for precommitted optimization. For any non-stationary policy $Q_*^{\sigma,\gamma}$0, there exist policies $Q_*^{\sigma,\gamma}$1 and $Q_*^{\sigma,\gamma}$2 such that

$Q_*^{\sigma,\gamma}$3

This one-step non-stationary representation is obtained by conditioning on the first action and next state: $Q_*^{\sigma,\gamma}$4 Since the tail is purely exponentially discounted, standard MDP theory yields a stationary tail policy. The optimal stationary component is exactly the optimal policy for the standard exponentially discounted MDP,

$Q_*^{\sigma,\gamma}$5

The optimal first-step action-value is

$Q_*^{\sigma,\gamma}$6

and the initial policy is greedy with respect to this hybrid action-value: $Q_*^{\sigma,\gamma}$7 An optimal policy $Q_*^{\sigma,\gamma}$8 exists and can be chosen deterministic (Eshwar, 7 Sep 2025).

3. Two-table committed Q-learning for quasi-hyperbolic agents

The quasi-hyperbolic control result implies that ordinary one-table Q-learning is insufficient. Standard Q-learning learns only the exponential optimal action-value function $Q_*^{\sigma,\gamma}$9. A committed QH agent, however, requires one value object for the tail policy after time $0$0 and a different value object for the initial choice. The relevant identity is

$0$1

which yields, after optimization,

$0$2

The algorithm proposed for committed control therefore maintains two iterates. $0$3 is intended to learn $0$4, and $0$5 is intended to learn $0$6. For each $0$7, sample

$0$8

observe $0$9, compute

$Q_*^\gamma$0

and update

$Q_*^\gamma$1

$Q_*^\gamma$2

In component form,

$Q_*^\gamma$3

$Q_*^\gamma$4

Action selection is explicitly time-indexed. At the initial step,

$Q_*^\gamma$5

while at all later steps,

$Q_*^\gamma$6

Committed Q-learning in this sense is therefore not simply “Q-learning with modified rewards”; it is a two-table scheme induced by the one-step non-stationary optimal structure.

The paper also gives a model-free off-policy evaluation method for a fixed one-step non-stationary policy $Q_*^\gamma$7. It maintains $Q_*^\gamma$8 for the stationary tail and $Q_*^\gamma$9 for the full policy, with coupled importance-weighted updates. Under Robbins-Monro stepsizes, bounded rewards, and bounded importance ratios, $t \ge 1$0 converge almost surely to $t \ge 1$1.

For control, the only assumptions are Robbins-Monro stepsizes and bounded rewards: $t \ge 1$2 Under these conditions, $t \ge 1$3 converge almost surely to

$t \ge 1$4

and the greedy pair $t \ge 1$5 is optimal. In the inventory control experiment with

$t \ge 1$6

the ground-truth optimal committed policy pair was an initial policy $t \ge 1$7 in states $t \ge 1$8 and a stationary policy $t \ge 1$9 in states $Q(z,\omega)$0; the model-free QH Q-learning algorithm recovered the same policy pair (Eshwar, 7 Sep 2025).

4. Reactive committed Q-learning under deterministic observations

In the reactive formulation, committed Q-learning is a variant of tabular Q-learning for partially observable control with deterministic observations, or equivalently hard state aggregation. The environment is finite and episodic, with latent states $Q(z,\omega)$1, terminal state $Q(z,\omega)$2, finite actions $Q(z,\omega)$3, and a deterministic observation map

$Q(z,\omega)$4

where $Q(z,\omega)$5. The agent observes only $Q(z,\omega)$6, not the true state $Q(z,\omega)$7.

The objective is not arbitrary history-dependent control but optimal reactive control. With a finite set of options $Q(z,\omega)$8, a reactive policy is

$Q(z,\omega)$9

and the goal is to maximize

$$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$0

The difficulty is that the aggregated observation process is generally non-Markov. Different hidden states inside the same feature class can have different continuation values and different transition effects, so a single value $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$1 may be ambiguous. This is the setting in which ordinary reactive Q-learning can fail.

Committed Q-learning changes the behavior process rather than the algebra of the one-step update. It maintains

$$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$2

initialized by

$$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$3

At the start of an episode, sample $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$4, set $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$5, and sample $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$6, where $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$7 is a fixed exploratory behavior policy. Then, for each step:

1. sample a primitive action $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$8;
2. transition to the next latent state and set $$M \equiv (\mathcal S,\mathcal A,\mathcal P,r,\sigma,\gamma),$$9;
3. compute

$\mathcal S,\mathcal A$0

1. update

$\mathcal S,\mathcal A$1

1. if $\mathcal S,\mathcal A$2, restart the episode;
2. if $\mathcal S,\mathcal A$3, resample $\mathcal S,\mathcal A$4;
3. if $\mathcal S,\mathcal A$5, keep the same option $\mathcal S,\mathcal A$6.

The commitment duration is therefore exactly “until the feature changes.” This is the defining step. Standard non-committed reactive Q-learning would resample an action or option at every time step, even if the feature stayed the same. Committed Q-learning instead treats a feature visit as a coherent segment. The corridor experiment illustrates the point: standard Q-learning fails to make progress even on short corridors, whereas Committed Q-learning rapidly converges to the optimal policy (Eberhard et al., 27 May 2026).

5. Quasi-Markov environments, rewiring, and convergence

The convergence theory for reactive committed Q-learning is built around quasi-Markov environments and rewire-robustness. A quasi-Markov environment is one in which each feature has a unique entrance distribution. Formally, with entrance space

$\mathcal S,\mathcal A$7

the quasi-Markov condition is equivalently

$\mathcal S,\mathcal A$8

The Markov property then holds whenever the feature changes. Inside a feature the process need not be Markov in the observation, but across feature transitions it is sufficiently structured to admit dynamic programming.

For a quasi-Markov environment $\mathcal S,\mathcal A$9, the paper defines an aggregate MDP $\mathcal P(\cdot\mid s,a)$0 on feature states $\mathcal P(\cdot\mid s,a)$1, with transition kernel

$\mathcal P(\cdot\mid s,a)$2

The key structural statement is the Entrance Value Lemma: $\mathcal P(\cdot\mid s,a)$3 Thus the value of a feature is the value of its entrance distribution. This yields an optimality equivalence between the latent quasi-Markov environment and the aggregate MDP.

A rewiring preserves the feature-level initial distribution, the feature-level transition law, and the within-feature latent dynamics, while changing entrance dynamics subject to a subspace condition. Rewire-robustness is then defined as follows: an environment is rewire-robust if any optimal policy of any rewiring of that environment is also optimal in the original environment. The paper proves the implication hierarchy

$\mathcal P(\cdot\mid s,a)$4

and

$\mathcal P(\cdot\mid s,a)$5

with all implications strict. In particular, rewire-robustness is strictly weaker than $\mathcal P(\cdot\mid s,a)$6-realizability.

The behavior-induced process is analyzed through the Markov chain

$\mathcal P(\cdot\mid s,a)$7

Under properness and full-support exploration, it is irreducible and has a unique stationary distribution $\mathcal P(\cdot\mid s,a)$8. The first convergence lemma shows that the iterates converge almost surely to $\mathcal P(\cdot\mid s,a)$9 satisfying

$\pi$00

The main theorem assumes that all policies are proper, that

$\pi$01

and that

$\pi$02

Under these conditions, the iterates of Committed Q-learning converge almost surely to a solution $\pi$03, with greedy policy

$\pi$04

If the environment is $\pi$05-rewire-robust, then $\pi$06 is optimal in the original environment. Since rewire-robustness implies $\pi$07-rewire-robustness, the paper concludes that Committed Q-learning converges almost surely to an optimal reactive policy under rewire-robustness (Eberhard et al., 27 May 2026).

6. Related formulations, boundaries, and recurrent misunderstandings

Several nearby papers are relevant only by analogy and clarify what committed Q-learning is not. "Learned Collusion" studies Qb-learning, a bias-parameterized family of Q-learning automata in which estimation remains standard but action selection is distorted by

$\pi$08

This is a persistent additive bias in the policy rule and is best interpreted as a commitment-like distortion, not as committed Q-learning in either of the main senses above (Compte, 2023).

"Lookahead-Bounded Q-Learning" constrains the updated value to remain within dynamic upper and lower bounds,

$\pi$09

This creates a commitment-like restriction on iterates, but the paper’s contribution is bounded or projection-based Q-learning using information-relaxation dual bounds, not commitment over actions, policies, or feature visits (Shar et al., 2020).

A second misconception is that convergence results for ordinary tabular Q-learning automatically extend to committed variants. Standard results require a finite discounted MDP, bounded rewards, infinite visitation of every state-action pair, and Robbins-Monro step sizes along each visited pair. Those results supply a baseline checklist, but they do not by themselves prove convergence for policy-coupled or behavior-committed variants (Regehr et al., 2021). Both main committed formulations therefore develop distinct fixed-point arguments: a two-function cascade for the quasi-hyperbolic case, and a Markovian-noise plus rewiring analysis for the reactive partially observable case.

A third misconception is that committed Q-learning is always a method for arbitrary partial observability. The reactive formulation optimizes over reactive policies $\pi$10 under deterministic observations and finite options, not over general memory-based POMDP policies. Conversely, the quasi-hyperbolic formulation is not a partial observability method at all; it is a control algorithm for time-inconsistent preferences.

Taken together, these formulations show that “committed Q-learning” is not a single algorithmic template but a family resemblance across settings where ordinary greedy, stationary, one-table Q-learning fails to capture the relevant control semantics. In one case the decisive object is the committed first step of a precommitted quasi-hyperbolic agent; in the other it is the temporally coherent feature visit induced by commitment to an option.