---
title: Recursive Backwards Q-Learning
url: https://www.emergentmind.com/topics/recursive-backwards-q-learning-rbql
type: topic
---

# Recursive Backwards Q-Learning

Recursive Backwards Q-Learning (RBQL) is a reinforcement learning algorithm designed for finite, deterministic, episodic Markov Decision Processes (MDPs). RBQL departs from model-free approaches such as Q-learning by explicitly building a transition and reward model during exploration and, after each episode, performing a recursive backward value propagation that sets each $Q(s,a)$ directly to its optimal value. This eliminates slow temporal-difference bootstrapping and dramatically accelerates convergence in deterministic domains, as demonstrated empirically in shortest-path maze benchmarks [2404.15822].

## 1. Deterministic Environment Model and Contrast to Model-Free Q-Learning

RBQL is formulated for settings where:
- $S$ (state set) and $A$ (action set) are finite;
- Transitions are deterministic: for each $(s,a)$, the next state $s'$ and reward $r$ are fixed;
- Episodes begin in a prescribed $s_0$ and terminate in one or more $s_T$, the sole sources of positive reward.

Standard Q-learning, in contrast, does not construct or exploit a model of $P$ or $R$, and updates $Q(s,a)$ incrementally at each step:
\[
Q(s,a)\;\leftarrow\;Q(s,a)\;+\;\alpha\Bigl(R(s,a)\;+\;\gamma\max_{a'}Q\bigl(s',a'\bigr)\;-\;Q(s,a)\Bigr)
\]
This results in slow per-episode propagation of terminal rewards, especially in large deterministic environments. RBQL, instead, builds $\widehat P$ (deterministic transitions) and $\widehat R$ (rewards) as it explores. When a terminal state is reached, it backpropagates value estimates via model inversion and a single backward sweep, directly setting all $Q$ values for the explored set to their fixed-point optimal values.

## 2. Formal Definitions and Update Equations

Let $\mathcal{M}=(S,A,P,R)$ denote the true MDP. RBQL maintains estimates:
- $\widehat P(s,a) \equiv s'$: the deterministic next state;
- $\widehat R(s,a) \equiv r$: the reward;

Upon reaching terminal state(s) $s_T$, RBQL assigns:
\[
\forall a\in A:\quad Q(s_T,a)\;=\;\widehat R(s_T,a)=0
\]
For any predecessor $(s,a)$, the recursive backup equation is
\begin{equation}
Q(s,a) = \widehat R(s,a) + \gamma\max_{a'\in A} Q(\widehat P(s,a), a')
\end{equation}

To propagate values, the algorithm inverts the transition model to construct the predecessor set:
\[
\mathrm{Pred}(s) = \{(s',a') \mid \widehat P(s',a') = s\}
\]
A breadth-first search (BFS) commencing from $s_T$ determines the update order, guaranteeing each $Q(s,a)$ is set only once per episode and always after all of its downstream dependencies are finalized.

## 3. Algorithmic Procedure

The central algorithmic workflow comprises two alternating phases: exploration/model-building and recursive backward value propagation. The following summarizes the key steps:

1. **Initialization**: All $\widehat P(s,a), \widehat R(s,a)$ marked "unknown"; all $Q(s,a)$ initialized to $0$.
2. **Exploration**: For each episode, the agent explores uncharted $(s,a)$ pairs or exploits greedily based on current $Q$ values. Exploration scheduling leverages $\epsilon$-decay.
3. **Model Update**: After executing $a$ in $s$, the observed $(s',r)$ are recorded in $\widehat P, \widehat R$.
4. **Episode Termination**: Once a terminal $s_T$ is reached, predecessors are computed by inverting $\widehat P$.
5. **Backward Sweep**: A BFS order is established from $s_T$ over all explored states/actions. For each $(s',a')$, $Q$ is updated according to the recursive backup equation.
6. **Policy Improvement**: Exploration flag and $\epsilon$ are decayed to balance exploration/exploitation.

No temporal-difference (TD) bootstrapping is performed; each $Q(s,a)$ is set to its global optimum with respect to the explored model after each episode.

## 4. Theoretical Properties and Convergence

- **Model Optimality**: The RBQL backward sweep computes the true optimal $Q(s,a)$ values with respect to the currently explored model; each $Q(s,a)$ is set exactly once per sweep, with no further adjustment needed until new $(s,a)$ pairs are discovered.
- **Policy Optimality**: Once all state–action pairs have been explored, RBQL reconstructs the globally optimal policy, as the single BFS backup is equivalent to dynamic programming value iteration on the discovered model.
- **Sample Complexity**: Each episode allows all explored states to be evaluated optimally. For a maze of shortest-path depth $d$, RBQL achieves solution in essentially one episode (plus exploration overhead), while standard Q-learning can require $O(d)$ episodes for equivalent credit propagation.
- **Computational Complexity**: Each backward sweep is $O(|S| + |A||S|)$ over the explored subset.
- **Convergence Dynamics**: Empirical evidence and theoretical argument demonstrate that the majority of learning occurs at the first terminal encounter; later episodes serve mainly to integrate newly discovered transitions or actions.

## 5. Empirical Evaluation: Maze Shortest-Path Task

Performance was measured on ensembles of $5\times5$, $10\times10$, and $15\times15$ random mazes, with each agent run for 25 episodes per maze (averaged over 50 random mazes). The principal metrics are average steps per episode and episode-over-episode improvement factors.

**Table 1:** Ratio of average steps (Q-Learning vs. RBQL) in episode 0 and episode 24.

| Grid     | QL steps (E0) | RBQL steps (E0) | Ratio (E0) | QL steps (E24) | RBQL steps (E24) | Ratio (E24) |
|----------|---------------|-----------------|------------|----------------|------------------|-------------|
| $5\times5$  | 278.06        | 191.84          | 1.45       | 49.14         | 9.62            | 5.11        |
| $10\times10$| 3308.46       | 843.52          | 3.92       | 281.44        | 23.68           | 11.89       |
| $15\times15$| 7180.98       | 1965.00         | 3.65       | 778.68        | 35.96           | 21.65       |

**Table 2:** Step count reduction factors from episode 0 to 24.

| Grid      | QL Step$_0$ | QL Step$_{24}$ | QL Factor | RBQL Step$_0$ | RBQL Step$_{24}$ | RBQL Factor |
|-----------|-------------|---------------|-----------|---------------|------------------|-------------|
| $5\times5$   | 278.06      | 49.14         | 5.66      | 191.84        | 9.62             | 19.94       |
| $10\times10$ | 3308.46     | 281.44        | 11.76     | 843.52        | 23.68            | 35.62       |
| $15\times15$ | 7180.98     | 778.68        | 9.22      | 1965.00       | 35.96            | 90.76       |

For the $10\times10$ grid, after 6 episodes RBQL nearly attains optimal step counts (close to the lower bound $2s-2$), whereas Q-learning still averages hundreds of steps after 25 episodes.

## 6. Practical Considerations: Advantages, Limitations, and Extensions

**Advantages**
- Accelerated value propagation: terminal reward is propagated instantly via a single backward sweep through the explored deterministic chain.
- All discovered $Q$ values are set optimally in each episode.
- Exploration focuses on as-yet-unseen actions, with greedy exploitation elsewhere.
- Requires orders of magnitude fewer episodes and steps to converge compared to standard Q-learning.

**Limitations**
- Applicability is restricted to deterministic transitions; for stochastic environments, direct adoption is not valid.
- RBQL targets episodic MDPs with finite horizons and clear terminal states; continuous or infinite-horizon tasks are unsupported.
- Requires storage of the full model ($\widehat P$, $\widehat R$) and incurs BFS computation per episode.

**Potential Extensions**
- Extension to stochastic MDPs by estimating transition probabilities $p(s'|s,a)$ and generalizing the backup equation:
  \[
  Q(s,a)\leftarrow \sum_{s'} p(s'|s,a)\left(R(s,a,s') + \gamma\max_{a'} Q(s',a')\right)
  \]
- Model compression via state aggregation to mitigate memory and computation requirements.
- Adaptation to multiple terminal states using either a dummy sink state or simultaneous backpropagation from all terminals.
- Hybridization with function approximation (e.g., "Deep RBQL") to address large or continuous state spaces.

## 7. Summary

Recursive Backwards Q-Learning replaces per-transition temporal-difference learning with per-episode, model-based dynamic programming on the explored state-action graph, achieving rapid convergence for deterministic, episodic MDPs. By constructing and inverting the explicit environment model, RBQL computes optimal $Q$ values for all encountered transitions in a single pass upon each episode termination. Empirical evidence demonstrates superior sample efficiency and convergence rates over standard Q-learning in grid-world maze tasks. The methodology is well-suited for deterministic, finite, episodic problems and offers clear potential for extension to more general environments [2404.15822].

Source: https://www.emergentmind.com/topics/recursive-backwards-q-learning-rbql