---
title: Regularized Agent-State Q-Learning
url: https://www.emergentmind.com/topics/regularized-agent-state-based-q-learning-rasql
type: topic
---

# Regularized Agent-State Q-Learning

Regularized Agent-State-Based Q-Learning (RASQL) is a reinforcement-learning framework for partially observable control in which the Q-update is indexed by a recursively updated **agent state** rather than by a belief state or other exact information state, and in which **policy regularization** is incorporated into the Bellman backup to encourage exploration and stabilize learning [2508.21314]. In the finite discounted POMDP setting, the framework formalizes a common practical pattern: observations and actions are compressed into an internal memory variable \(z_t\), value learning is performed on \((z_t,a_t)\), and the resulting algorithm converges under standard stochastic-approximation and ergodicity assumptions. The central caveat is equally important: the limit is generally **not** the optimal Q-function of the original POMDP, but the fixed point of an induced regularized MDP on the agent-state space, and that induced MDP depends on the stationary distribution generated by the behavior policy [2508.21314].

## 1. Formal setting and the notion of agent state

RASQL is posed on a finite discounted POMDP with environment state \(s_t \in \mathcal S\), action \(a_t \in \mathcal A\), observation \(y_t \in \mathcal Y\), discount factor \(\gamma \in [0,1)\), and reward
\[
r_t = r(s_t,a_t) \in [0,R_{\max}].
\]
The dynamics satisfy
\[
P(s_{t+1}, y_{t+1} \mid s_{1:t}, y_{1:t}, a_{1:t}) = P(s_{t+1}, y_{t+1} \mid s_t, a_t),
\]
with initial state \(s_1 \sim \rho\). A general history-dependent policy is written as \(\vec{\boldsymbol{\pi}} = (\vec\pi_1,\vec\pi_2,\ldots)\), where
\[
a_t \sim \vec\pi_t(y_{1:t}, a_{1:t-1}),
\]
and its discounted return is
\[
J_{\vec{\boldsymbol{\pi}}} \coloneqq E^{\vec{\boldsymbol{\pi}}}\!\left[ \sum_{t=1}^\infty \gamma^{t-1} r(s_t,a_t) \;\middle|\; s_1 \sim \rho \right].
\]
The control objective is to maximize this return [2508.21314].

The distinctive state variable in RASQL is the **agent state**. Rather than assuming access to a belief state, the learner maintains a model-free recursively updateable internal state \(z_t \in \mathcal Z\) via
\[
z_{t+1} = \phi(z_t, y_{t+1}, a_t).
\]
This construction captures finite-memory controllers, frame stacking, and finite-state controllers, and abstracts the role played by recurrent hidden states in practical deep RL [2508.21314]. The broader agent-state literature emphasizes that such a \(z_t\) is merely a recursively updateable compression of history, not necessarily an information state. In particular, a belief state is model-dependent and therefore typically unavailable in model-free RL, whereas an agent state is model-free but may fail to satisfy the Markov property needed for standard dynamic programming arguments [2409.15703].

That distinction is foundational. If \(z_t\) were an information state, standard MDP RL theory would apply directly. In RASQL, \(z_t\) need not be sufficient for control. The algorithm therefore operates in a regime that mirrors practical recurrent Q-learning, but its limiting object must be characterized through an induced surrogate control problem rather than through the Bellman equations of the original POMDP [2508.21314].

## 2. Regularized Bellman backup and the induced regularized MDP

RASQL augments agent-state Q-learning with a strongly convex policy regularizer
\[
\Omega : \Delta(\mathcal A) \to \mathbb R.
\]
The framework uses the regularized-MDP machinery associated with the convex conjugate \(\Omega^*\). The paper reviews two canonical cases. For entropy regularization,
\[
\Omega(p) = \frac{1}{\beta}\sum_{a \in \mathcal A} p(a)\ln p(a),
\qquad
\Omega^*(q) = \frac{1}{\beta} \ln\left(\sum_{a\in\mathcal A} e^{\beta q(a)}\right),
\]
and the induced policy is
\[
p^\star(a) = \frac{e^{\beta q(a)}}{\sum_{a'} e^{\beta q(a')}}.
\]
For KL regularization,
\[
\Omega(p) = \frac{1}{\beta}\sum_{a\in\mathcal A} p(a)\ln\frac{p(a)}{p_{\rm ref}(a)},
\]
\[
\Omega^*(q) = \frac{1}{\beta} \ln\left( \sum_{a\in\mathcal A} p_{\rm ref}(a)e^{\beta q(a)} \right),
\]
and
\[
p^\star(a) = \frac{p_{\rm ref}(a)e^{\beta q(a)}}{\sum_{a'}p_{\rm ref}(a')e^{\beta q(a')}}.
\]
These are standard regularized-control constructions in which the hard \(\max\) backup is replaced by a smooth regularized value map [2508.21314]. This places RASQL within the broader family of entropy- and KL-regularized value-based methods in which the Bellman operator is modified by a policy regularizer rather than by an unconstrained maximization [2108.07041].

The behavior policy is a fixed stationary agent-state-based exploration policy,
\[
a_t \sim \pi_{\rm expl}(\cdot \mid z_t).
\]
The current regularized greedy policy induced by \(Q_t\) is
\[
\pi_t(\cdot \mid z) = \argmax_{\xi \in \Delta(\mathcal A)} \left\{ \sum_{a\in\mathcal A}\xi(a)Q_t(z,a)-\Omega(\xi) \right\}.
\]
The tabular RASQL update is
\[
Q_{t+1}(z,a) = Q_t(z,a) + \alpha_t(z,a) \left[ r_t + \gamma \,\Omega^*(Q_t(z_{t+1},\cdot)) - Q_t(z,a) \right],
\]
with \(\alpha_t(z,a)=0\) whenever \((z,a)\neq (z_t,a_t)\) [2508.21314].

The key theoretical construction is the **induced regularized MDP** on the agent-state space \(\mathcal Z\). Under the stationary behavior policy, assume the joint process \(\{(S_t,Y_t,Z_t,A_t)\}_{t\ge 1}\) converges to a limiting distribution \(\zeta_{\rm expl}\). Then the induced one-step reward is
\[
r_{\pi_{\rm expl}}(z,a) \coloneqq \sum_{s\in\mathcal S} r(s,a)\,\zeta_{\rm expl}(s\mid z),
\]
and the induced transition kernel is
\[
P_{\pi_{\rm expl}}(z' \mid z,a) \coloneqq \sum_{(s,y')\in\mathcal S\times\mathcal Y} 1_{\{z'=\phi(z,y',a)\} \,P(y'\mid s,a)\, \zeta_{\rm expl}(s\mid z).
\]
Because
\[
\zeta_{\rm expl}(s,z,y,a) = \zeta_{\rm expl}(s,z)\,\pi_{\rm expl}(a\mid z)\,P(y\mid s,a),
\]
one has
\[
\zeta_{\rm expl}(s\mid z,a)=\zeta_{\rm expl}(s\mid z).
\]
The regularized Bellman operator of the induced MDP is therefore
\[
(\mathcal T_{\pi_{\rm expl}}^\Omega Q)(z,a) = r_{\pi_{\rm expl}}(z,a) + \gamma \sum_{z'\in\mathcal Z} P_{\pi_{\rm expl}}(z'\mid z,a)\, \Omega^*(Q(z',\cdot)),
\]
with fixed point
\[
Q_{\pi_{\rm expl}}(z,a) = r_{\pi_{\rm expl}}(z,a) + \gamma \sum_{z'\in\mathcal Z} P_{\pi_{\rm expl}}(z'\mid z,a)\, \Omega^*(Q_{\pi_{\rm expl}}(z',\cdot)).
\]
This fixed point is the asymptotic target of RASQL [2508.21314].

## 3. Convergence theory and the meaning of the limit

The convergence theorem assumes standard asynchronous stochastic-approximation conditions. For every \((z,a)\), the step sizes are measurable with respect to the history of visited agent-state/action pairs, satisfy
\[
\alpha_t(z,a)=0 \quad \text{if } (z,a)\neq (z_t,a_t),
\]
and
\[
\sum_{t\ge 1}\alpha_t(z,a)=\infty, \qquad \sum_{t\ge 1}\alpha_t(z,a)^2<\infty
\quad \text{a.s.}
\]
It also assumes that the behavior policy induces a limiting distribution \(\zeta_{\rm expl}\) for \(\{(S_t,Y_t,Z_t,A_t)\}_{t\ge 1}\) and that every \((z,a)\) is visited infinitely often [2508.21314].

Under these assumptions, RASQL converges almost surely:
\[
Q_t \to Q_{\pi_{\rm expl}} \qquad \text{a.s.}
\]
The limit is the unique fixed point of the induced regularized Bellman equation on \(\mathcal Z \times \mathcal A\) [2508.21314]. Uniqueness follows from the contraction property of the regularized Bellman operator for the induced finite-state regularized MDP.

The interpretive content of this theorem is more significant than the bare convergence claim. The result does **not** identify \(Q_{\pi_{\rm expl}}\) with the optimal Q-function of the original POMDP, nor with the optimal Q-function over all history-dependent policies, nor even with the optimal Q-function over all agent-state-based policies. Instead, the theorem says that tabular regularized Q-learning with an arbitrary recursively updated agent state solves a **behavior-policy-dependent averaged control problem** on the agent-state space. The limiting reward and transition model are obtained by conditioning on the stationary hidden-state distribution given the current agent state under the behavior policy. In that sense, RASQL is off-policy in sampling, but its target fixed point is still shaped by the behavior policy’s stationary occupancy measure [2508.21314].

The proof uses asynchronous stochastic approximation with Markov noise. The error is written as
\[
\Delta_{t+1} \coloneqq Q_{t+1} - Q_{\pi_{\rm expl}},
\]
and decomposed into terms corresponding to reward sampling noise, transition sampling noise, and the nonlinear regularized-backup discrepancy. The hidden-state non-Markovianity of \(Z_t\) is handled by passing to the stationary conditional law induced by the behavior policy, rather than by asserting that \(Z_t\) itself is an information state. Regularization enters the stability argument through the bound on the nonlinear term
\[
\gamma\Omega^*(Q_t(Z_{t+1},\cdot)) - \gamma\Omega^*(Q_{\pi_{\rm expl}}(Z_{t+1},\cdot)),
\]
which is controlled by the sup-norm difference between current and limiting Q-functions [2508.21314].

## 4. Periodic RASQL and non-stationary agent-state policies

The periodic extension, RePASQL, addresses the case in which stationary agent-state policies are too restrictive. The algorithm maintains a phase-indexed family of Q-tables
\[
Q_t^\ell(z,a), \qquad \ell \in [L] = \{0,\dots,L-1\},
\]
and updates only the component corresponding to the current phase \(\llbracket t \rrbracket = t \bmod L\):
\[
Q_{t+1}^{\ell}(z,a) = Q_t^{\ell}(z,a) + \alpha_t^\ell(z,a) \left[ r_t + \gamma \Omega^*(Q_t^{\llbracket \ell+1\rrbracket}(z_{t+1},\cdot)) - Q_t^\ell(z,a) \right].
\]
The behavior policy is itself periodic,
\[
\pi_{\rm expl}=\{\pi_{\rm expl}^\ell\}_{\ell\in[L]},
\]
and the induced limiting distribution is phase dependent, denoted \(\zeta_{\rm expl}^\ell\) [2508.21314].

For each phase \(\ell\), the induced reward and transition are
\[
r_{\pi_{\rm expl}^\ell}(z,a) \coloneqq \sum_{s\in\mathcal S} r(s,a)\zeta_{\rm expl}^\ell(s\mid z),
\]
and
\[
P_{\pi_{\rm expl}^\ell}(z'\mid z,a) \coloneqq \sum_{(s,y')\in\mathcal S\times\mathcal Y} 1_{\{z'=\phi(z,y',a)\} P(y'\mid s,a)\zeta_{\rm expl}^\ell(s\mid z).
\]
This yields a family of phase-wise regularized Bellman operators
\[
\mathcal T_\Omega^\ell Q(z,a) = r_{\pi_{\rm expl}^\ell}(z,a) + \gamma \sum_{z'\in\mathcal Z} P_{\pi_{\rm expl}^\ell}(z'\mid z,a)\Omega^*(Q(z',\cdot)),
\]
and the cyclic composition
\[
\mathcal T_{\Omega,\mathrm{per}}^\ell = \mathcal T_\Omega^\ell \mathcal T_\Omega^{\llbracket \ell+1\rrbracket} \cdots \mathcal T_\Omega^{\llbracket \ell+L-1\rrbracket}.
\]
The resulting periodic fixed-point equations are
\[
Q_{\pi_{\rm expl}^\ell}(z,a) = r_{\pi_{\rm expl}^\ell}(z,a) + \gamma \sum_{z'\in\mathcal Z} P_{\pi_{\rm expl}^\ell}(z'\mid z,a) V_{\pi_{\rm expl}^{\llbracket \ell+1\rrbracket}}(z'),
\]
with
\[
V_{\pi_{\rm expl}^\ell}(z)=\Omega^*(Q_{\pi_{\rm expl}^\ell}(z,\cdot)).
\]
Under periodic analogues of the stationary assumptions, RePASQL converges almost surely to the unique family \(\{Q_{\pi_{\rm expl}^\ell}\}_{\ell\in[L]}\) [2508.21314].

This periodic extension is motivated by the broader observation that when the agent state is not Markov, non-stationary agent-state-based policies can outperform stationary ones. Periodic agent-state-based Q-learning was introduced precisely to exploit this fact, and its convergence target is a periodic surrogate MDP rather than a stationary one [2407.06121]. A plausible implication is that regularization and periodicity address different deficiencies: regularization smooths and stabilizes the control update, whereas periodicity enlarges the admissible policy class when the agent-state representation is insufficient.

## 5. Position within the broader literature

RASQL sits at the intersection of two strands of reinforcement-learning theory. The first is the agent-state literature for POMDPs. In that line, the central abstraction is the recursively updateable internal state
\[
Z_{t+1} = \phi(Z_t, Y_{t+1}, A_t),
\]
which generalizes frame stacking, finite-state controllers, and recurrent memories. That literature emphasizes that an agent state is not generally an information state, that stationary policies may be suboptimal when the representation is non-Markovian, and that approximate information states provide a principled way to assess representation quality [2409.15703]. RASQL adopts the same agent-state abstraction, but adds policy regularization to the Bellman backup rather than leaving the update unregularized [2508.21314].

The second strand is regularized value-based RL. In entropy- and KL-regularized control, the hard maximization in the Bellman backup is replaced by a regularized value operator, typically producing stochastic soft-greedy policies and smoother fixed-point equations. RASQL uses precisely this regularized-control template, but applies it on the induced MDP over agent states rather than on a true fully observed state space [2508.21314]. This makes it conceptually adjacent to regularized Bellman methods that interpret entropy or KL penalties as part of the control objective, including formulations in which the policy/Q relation is enforced through entropy- or KL-regularized dynamic programming [2108.07041].

A useful contrast concerns **what is being regularized**. In RASQL, the regularizer acts on the policy used in the Bellman backup. In recurrent Q-learning with approximate information state losses, by contrast, the regularization target is the **representation** itself: the recurrent state is trained to predict reward and future compressed state or observation, and lower AIS loss correlates strongly with better control performance [2306.05991]. This suggests two distinct but compatible design axes for partially observable Q-learning: regularizing the control objective, as in RASQL, and regularizing the agent-state representation toward approximate sufficiency, as in AIS-based recurrent Q-learning.

## 6. Interpretation, limitations, and empirical illustration

Several misconceptions are explicitly ruled out by the RASQL analysis. First, convergence does **not** imply that the original POMDP has been solved. The learned object is the fixed point of an induced regularized MDP defined by \(\zeta_{\rm expl}(s\mid z)\), not the optimal history-based Q-function of the hidden-state process [2508.21314]. Second, the result is not behavior-policy invariant. Because both
\[
r_{\pi_{\rm expl}}(z,a)
\quad \text{and} \quad
P_{\pi_{\rm expl}}(z'\mid z,a)
\]
depend on the stationary distribution generated by \(\pi_{\rm expl}\), changing the behavior policy changes the target fixed point. Third, regularization does not repair an inadequate agent-state representation. If the update rule \(\phi\) loses control-relevant information, the induced MDP can remain a poor proxy for the underlying POMDP even though the stochastic approximation converges cleanly.

These limitations align with the broader agent-state perspective. When the internal state is not an information state, stationary deterministic policies can be intrinsically suboptimal, and stochastic or non-stationary policies can outperform them [2409.15703]. RASQL partially addresses this by allowing stochastic regularized greedy policies; RePASQL goes further by allowing periodic non-stationarity [2508.21314].

The numerical examples in the RASQL paper are designed to validate the convergence characterization rather than to demonstrate global optimality. In the stationary experiment, the POMDP has \(\mathcal S=\{0,1,2,3\}\), \(\mathcal A=\{0,1\}\), \(\mathcal Y=\{0,1\}\), \(\gamma=0.9\), initial distribution
\[
\rho = [0.3,\;0.0,\;0.2,\;0.5],
\]
and the agent state is chosen to be the observation itself,
\[
z_t = y_t.
\]
With behavior policy
\[
\pi_{\rm expl}(a\mid z) =
\begin{bmatrix}
0.2 & 0.8\\
0.8 & 0.2
\end{bmatrix},
\]
\(25\) random seeds, \(10^5\) iterations, and regularization coefficient \(\beta=1.0\), the empirical \(Q_t(z,a)\) trajectories converge to the theoretical limit \(Q_{\pi_{\rm expl}}(z,a)\) predicted by the induced-MDP fixed-point equation [2508.21314].

In the periodic experiment, the same observation-based agent state \(z_t=y_t\) is used, but with period \(L=2\) and periodic behavior policy
\[
\pi_{\rm expl}^{0}(a\mid z) =
\begin{bmatrix}
0.2 & 0.8\\
0.8 & 0.2
\end{bmatrix},
\qquad
\pi_{\rm expl}^{1}(a\mid z) =
\begin{bmatrix}
0.8 & 0.2\\
0.2 & 0.8
\end{bmatrix}.
\]
Again, across \(25\) random seeds and \(10^5\) iterations, the phase-specific trajectories converge to the predicted limits \(Q_{\pi_{\rm expl}^\ell}(z,a)\) [2508.21314]. These experiments reinforce the main theoretical message: empirical convergence behavior matches the induced regularized MDP, and the asymptotic solution is shaped jointly by the agent-state representation and the stationary or periodic exploration policy.

In summary, RASQL provides a precise theoretical language for a large class of practical partially observable Q-learning algorithms. Its contribution is not an optimality theorem for POMDPs, but a convergence theorem with a sharp identification of the limit: regularized Q-learning on a non-sufficient agent state converges to the fixed point of a behavior-policy-dependent regularized surrogate MDP on that agent-state space [2508.21314].

Source: https://www.emergentmind.com/topics/regularized-agent-state-based-q-learning-rasql