Regularized Agent-State Q-Learning

• Regularized Agent-State Q-Learning is a framework that leverages recursively updated agent states and a convex policy regularizer to guide Q-learning in partially observable settings.
• It approximates control by compressing observations and actions into an internal state, converging to a behavior-policy-dependent fixed point instead of the optimal POMDP Q-function.
• The periodic extension, RePASQL, uses phase-indexed Q-tables to handle non-stationary dynamics, enhancing stability and performance in challenging environments.

Regularized Agent-State-Based Q-Learning (RASQL) is a reinforcement-learning framework for partially observable control in which the Q-update is indexed by a recursively updated agent state rather than by a belief state or other exact information state, and in which policy regularization is incorporated into the Bellman backup to encourage exploration and stabilize learning (Sinha et al., 29 Aug 2025). In the finite discounted POMDP setting, the framework formalizes a common practical pattern: observations and actions are compressed into an internal memory variable $z_t$, value learning is performed on $(z_t,a_t)$, and the resulting algorithm converges under standard stochastic-approximation and ergodicity assumptions. The central caveat is equally important: the limit is generally not the optimal Q-function of the original POMDP, but the fixed point of an induced regularized MDP on the agent-state space, and that induced MDP depends on the stationary distribution generated by the behavior policy (Sinha et al., 29 Aug 2025).

1. Formal setting and the notion of agent state

RASQL is posed on a finite discounted POMDP with environment state $s_t \in \mathcal S$, action $a_t \in \mathcal A$, observation $y_t \in \mathcal Y$, discount factor $\gamma \in [0,1)$, and reward

$r_t = r(s_t,a_t) \in [0,R_{\max}].$

The dynamics satisfy

$$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$

with initial state $s_1 \sim \rho$. A general history-dependent policy is written as $$\vec{\boldsymbol{\pi}} = (\vec\pi_1,\vec\pi_2,\ldots)$$, where

$(z_t,a_t)$0

and its discounted return is

$(z_t,a_t)$1

The control objective is to maximize this return (Sinha et al., 29 Aug 2025).

The distinctive state variable in RASQL is the agent state. Rather than assuming access to a belief state, the learner maintains a model-free recursively updateable internal state $(z_t,a_t)$2 via

$(z_t,a_t)$3

This construction captures finite-memory controllers, frame stacking, and finite-state controllers, and abstracts the role played by recurrent hidden states in practical deep RL (Sinha et al., 29 Aug 2025). The broader agent-state literature emphasizes that such a $(z_t,a_t)$4 is merely a recursively updateable compression of history, not necessarily an information state. In particular, a belief state is model-dependent and therefore typically unavailable in model-free RL, whereas an agent state is model-free but may fail to satisfy the Markov property needed for standard dynamic programming arguments (Sinha et al., 2024).

That distinction is foundational. If $(z_t,a_t)$5 were an information state, standard MDP RL theory would apply directly. In RASQL, $(z_t,a_t)$6 need not be sufficient for control. The algorithm therefore operates in a regime that mirrors practical recurrent Q-learning, but its limiting object must be characterized through an induced surrogate control problem rather than through the Bellman equations of the original POMDP (Sinha et al., 29 Aug 2025).

2. Regularized Bellman backup and the induced regularized MDP

RASQL augments agent-state Q-learning with a strongly convex policy regularizer

$(z_t,a_t)$7

The framework uses the regularized-MDP machinery associated with the convex conjugate $(z_t,a_t)$8. The paper reviews two canonical cases. For entropy regularization,

$(z_t,a_t)$9

and the induced policy is

$s_t \in \mathcal S$0

For KL regularization,

$s_t \in \mathcal S$1

$s_t \in \mathcal S$2

and

$s_t \in \mathcal S$3

These are standard regularized-control constructions in which the hard $s_t \in \mathcal S$4 backup is replaced by a smooth regularized value map (Sinha et al., 29 Aug 2025). This places RASQL within the broader family of entropy- and KL-regularized value-based methods in which the Bellman operator is modified by a policy regularizer rather than by an unconstrained maximization (Vieillard et al., 2021).

The behavior policy is a fixed stationary agent-state-based exploration policy,

$s_t \in \mathcal S$5

The current regularized greedy policy induced by $s_t \in \mathcal S$6 is

$s_t \in \mathcal S$7

The tabular RASQL update is

$s_t \in \mathcal S$8

with $s_t \in \mathcal S$9 whenever $a_t \in \mathcal A$0 (Sinha et al., 29 Aug 2025).

The key theoretical construction is the induced regularized MDP on the agent-state space $a_t \in \mathcal A$1. Under the stationary behavior policy, assume the joint process $a_t \in \mathcal A$2 converges to a limiting distribution $a_t \in \mathcal A$3. Then the induced one-step reward is

$a_t \in \mathcal A$4

and the induced transition kernel is

$a_t \in \mathcal A$5

Because

$a_t \in \mathcal A$6

one has

$a_t \in \mathcal A$7

The regularized Bellman operator of the induced MDP is therefore

$a_t \in \mathcal A$8

with fixed point

$a_t \in \mathcal A$9

This fixed point is the asymptotic target of RASQL (Sinha et al., 29 Aug 2025).

3. Convergence theory and the meaning of the limit

The convergence theorem assumes standard asynchronous stochastic-approximation conditions. For every $y_t \in \mathcal Y$0, the step sizes are measurable with respect to the history of visited agent-state/action pairs, satisfy

$y_t \in \mathcal Y$1

and

$y_t \in \mathcal Y$2

It also assumes that the behavior policy induces a limiting distribution $y_t \in \mathcal Y$3 for $y_t \in \mathcal Y$4 and that every $y_t \in \mathcal Y$5 is visited infinitely often (Sinha et al., 29 Aug 2025).

Under these assumptions, RASQL converges almost surely: $y_t \in \mathcal Y$6 The limit is the unique fixed point of the induced regularized Bellman equation on $y_t \in \mathcal Y$7 (Sinha et al., 29 Aug 2025). Uniqueness follows from the contraction property of the regularized Bellman operator for the induced finite-state regularized MDP.

The interpretive content of this theorem is more significant than the bare convergence claim. The result does not identify $y_t \in \mathcal Y$8 with the optimal Q-function of the original POMDP, nor with the optimal Q-function over all history-dependent policies, nor even with the optimal Q-function over all agent-state-based policies. Instead, the theorem says that tabular regularized Q-learning with an arbitrary recursively updated agent state solves a behavior-policy-dependent averaged control problem on the agent-state space. The limiting reward and transition model are obtained by conditioning on the stationary hidden-state distribution given the current agent state under the behavior policy. In that sense, RASQL is off-policy in sampling, but its target fixed point is still shaped by the behavior policy’s stationary occupancy measure (Sinha et al., 29 Aug 2025).

The proof uses asynchronous stochastic approximation with Markov noise. The error is written as

$y_t \in \mathcal Y$9

and decomposed into terms corresponding to reward sampling noise, transition sampling noise, and the nonlinear regularized-backup discrepancy. The hidden-state non-Markovianity of $\gamma \in [0,1)$0 is handled by passing to the stationary conditional law induced by the behavior policy, rather than by asserting that $\gamma \in [0,1)$1 itself is an information state. Regularization enters the stability argument through the bound on the nonlinear term

$\gamma \in [0,1)$2

which is controlled by the sup-norm difference between current and limiting Q-functions (Sinha et al., 29 Aug 2025).

4. Periodic RASQL and non-stationary agent-state policies

The periodic extension, RePASQL, addresses the case in which stationary agent-state policies are too restrictive. The algorithm maintains a phase-indexed family of Q-tables

$\gamma \in [0,1)$3

and updates only the component corresponding to the current phase $\gamma \in [0,1)$4: $\gamma \in [0,1)$5 The behavior policy is itself periodic,

$\gamma \in [0,1)$6

and the induced limiting distribution is phase dependent, denoted $\gamma \in [0,1)$7 (Sinha et al., 29 Aug 2025).

For each phase $\gamma \in [0,1)$8, the induced reward and transition are

$\gamma \in [0,1)$9

and

$r_t = r(s_t,a_t) \in [0,R_{\max}].$0

This yields a family of phase-wise regularized Bellman operators

$r_t = r(s_t,a_t) \in [0,R_{\max}].$1

and the cyclic composition

$r_t = r(s_t,a_t) \in [0,R_{\max}].$2

The resulting periodic fixed-point equations are

$r_t = r(s_t,a_t) \in [0,R_{\max}].$3

with

$r_t = r(s_t,a_t) \in [0,R_{\max}].$4

Under periodic analogues of the stationary assumptions, RePASQL converges almost surely to the unique family $r_t = r(s_t,a_t) \in [0,R_{\max}].$5 (Sinha et al., 29 Aug 2025).

This periodic extension is motivated by the broader observation that when the agent state is not Markov, non-stationary agent-state-based policies can outperform stationary ones. Periodic agent-state-based Q-learning was introduced precisely to exploit this fact, and its convergence target is a periodic surrogate MDP rather than a stationary one (Sinha et al., 2024). A plausible implication is that regularization and periodicity address different deficiencies: regularization smooths and stabilizes the control update, whereas periodicity enlarges the admissible policy class when the agent-state representation is insufficient.

5. Position within the broader literature

RASQL sits at the intersection of two strands of reinforcement-learning theory. The first is the agent-state literature for POMDPs. In that line, the central abstraction is the recursively updateable internal state

$r_t = r(s_t,a_t) \in [0,R_{\max}].$6

which generalizes frame stacking, finite-state controllers, and recurrent memories. That literature emphasizes that an agent state is not generally an information state, that stationary policies may be suboptimal when the representation is non-Markovian, and that approximate information states provide a principled way to assess representation quality (Sinha et al., 2024). RASQL adopts the same agent-state abstraction, but adds policy regularization to the Bellman backup rather than leaving the update unregularized (Sinha et al., 29 Aug 2025).

The second strand is regularized value-based RL. In entropy- and KL-regularized control, the hard maximization in the Bellman backup is replaced by a regularized value operator, typically producing stochastic soft-greedy policies and smoother fixed-point equations. RASQL uses precisely this regularized-control template, but applies it on the induced MDP over agent states rather than on a true fully observed state space (Sinha et al., 29 Aug 2025). This makes it conceptually adjacent to regularized Bellman methods that interpret entropy or KL penalties as part of the control objective, including formulations in which the policy/Q relation is enforced through entropy- or KL-regularized dynamic programming (Vieillard et al., 2021).

A useful contrast concerns what is being regularized. In RASQL, the regularizer acts on the policy used in the Bellman backup. In recurrent Q-learning with approximate information state losses, by contrast, the regularization target is the representation itself: the recurrent state is trained to predict reward and future compressed state or observation, and lower AIS loss correlates strongly with better control performance (Seyedsalehi et al., 2023). This suggests two distinct but compatible design axes for partially observable Q-learning: regularizing the control objective, as in RASQL, and regularizing the agent-state representation toward approximate sufficiency, as in AIS-based recurrent Q-learning.

6. Interpretation, limitations, and empirical illustration

Several misconceptions are explicitly ruled out by the RASQL analysis. First, convergence does not imply that the original POMDP has been solved. The learned object is the fixed point of an induced regularized MDP defined by $r_t = r(s_t,a_t) \in [0,R_{\max}].$7, not the optimal history-based Q-function of the hidden-state process (Sinha et al., 29 Aug 2025). Second, the result is not behavior-policy invariant. Because both

$r_t = r(s_t,a_t) \in [0,R_{\max}].$8

depend on the stationary distribution generated by $r_t = r(s_t,a_t) \in [0,R_{\max}].$9, changing the behavior policy changes the target fixed point. Third, regularization does not repair an inadequate agent-state representation. If the update rule $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$0 loses control-relevant information, the induced MDP can remain a poor proxy for the underlying POMDP even though the stochastic approximation converges cleanly.

These limitations align with the broader agent-state perspective. When the internal state is not an information state, stationary deterministic policies can be intrinsically suboptimal, and stochastic or non-stationary policies can outperform them (Sinha et al., 2024). RASQL partially addresses this by allowing stochastic regularized greedy policies; RePASQL goes further by allowing periodic non-stationarity (Sinha et al., 29 Aug 2025).

The numerical examples in the RASQL paper are designed to validate the convergence characterization rather than to demonstrate global optimality. In the stationary experiment, the POMDP has $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$1, $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$2, $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$3, $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$4, initial distribution

$$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$5

and the agent state is chosen to be the observation itself,

$$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$6

With behavior policy

$$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$7

$$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$8 random seeds, $$P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),$$9 iterations, and regularization coefficient $s_1 \sim \rho$0, the empirical $s_1 \sim \rho$1 trajectories converge to the theoretical limit $s_1 \sim \rho$2 predicted by the induced-MDP fixed-point equation (Sinha et al., 29 Aug 2025).

In the periodic experiment, the same observation-based agent state $s_1 \sim \rho$3 is used, but with period $s_1 \sim \rho$4 and periodic behavior policy

$s_1 \sim \rho$5

Again, across $s_1 \sim \rho$6 random seeds and $s_1 \sim \rho$7 iterations, the phase-specific trajectories converge to the predicted limits $s_1 \sim \rho$8 (Sinha et al., 29 Aug 2025). These experiments reinforce the main theoretical message: empirical convergence behavior matches the induced regularized MDP, and the asymptotic solution is shaped jointly by the agent-state representation and the stationary or periodic exploration policy.

In summary, RASQL provides a precise theoretical language for a large class of practical partially observable Q-learning algorithms. Its contribution is not an optimality theorem for POMDPs, but a convergence theorem with a sharp identification of the limit: regularized Q-learning on a non-sufficient agent state converges to the fixed point of a behavior-policy-dependent regularized surrogate MDP on that agent-state space (Sinha et al., 29 Aug 2025).