Quantum AIXI: A Feasible RL Framework

• Quantum AIXI is a reinforcement learning framework that approximates the ideal AIXI model via efficient Monte Carlo Tree Search and Bayesian sequence prediction.
• It employs ρ-UCT planning and action-conditional Context Tree Weighting to manage uncertainties in partially observable, stochastic environments.
• Empirical studies show that this approach achieves near-optimal performance with scalable computational costs across diverse domains.

MC-AIXI-CTW is a computationally feasible, general reinforcement learning agent that directly approximates the AIXI model, which is a Bayesian optimality notion for reinforcement learning in arbitrary computable environments. The MC-AIXI-CTW agent employs a synergy of Monte Carlo Tree Search (MCTS), specifically a history-based variant of UCT ("ρ-UCT"), and an agent-specific extension of the Context Tree Weighting (CTW) algorithm to approximate the expectimax value function and the Solomonoff mixture, respectively. This architecture enables the agent to achieve strong empirical performance in a variety of partially observable, stochastic, and unknown domains, while maintaining polynomial resource usage in tree depth, planning horizon, and number of simulations (Veness et al., 2010, 0909.0801).

1. AIXI Background and Motivation

AIXI formalizes the reinforcement learning problem as a sequential decision process where, at each time step, the agent selects an action $a_t \in \mathcal{A}$, receives a percept $x_t = (o_t, r_t)$ with observation $o_t$ and reward $r_t$, and seeks to maximize total future reward. AIXI defines the optimal policy by a Bayesian expectimax over all computable environment models, using Solomonoff's universal prior:

$$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$

where

$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$

with the sum over all computable semimeasures $\nu$ and $K(\nu)$ the Kolmogorov complexity. This ideal is computationally infeasible due to the incomputability of Solomonoff induction and the exponential complexity of expectimax planning (0909.0801).

2. Algorithmic Structure: ρ-UCT Planning

MC-AIXI-CTW approximates the AIXI expectimax with ρ-UCT, a MCTS variant operating on histories rather than state space. The core simulation loop includes selection, expansion, simulation (rollout), and backpropagation:

• Selection: From the current history $h$, select actions recursively using the UCB1 formula:

$$a^* = \arg\max_{a \in \mathcal{A}} \left[ \frac{1}{m(\beta - \alpha)} \hat{V}(ha) + C \sqrt{\frac{\ln T(h)}{T(ha)}} \right]$$

where $x_t = (o_t, r_t)$0 is the current value estimate, $x_t = (o_t, r_t)$1 is the visit count of node $x_t = (o_t, r_t)$2, $x_t = (o_t, r_t)$3 normalizes the reward, $x_t = (o_t, r_t)$4 is the exploration coefficient, and $x_t = (o_t, r_t)$5 is the planning horizon (Veness et al., 2010, 0909.0801).

• Expansion: On reaching an unvisited node or a chance node (after an action), sample a percept and expand the tree.
• Simulation (Rollout): From a new decision node or at horizon, select a random policy (typically uniform random), execute to terminal depth, and accumulate rewards.
• Backpropagation: After each simulation, propagate the sampled reward up the visited path:

$x_t = (o_t, r_t)$6

$x_t = (o_t, r_t)$7

This anytime planner enables MC-AIXI-CTW to concentrate computational effort on the most promising parts of the search tree (Veness et al., 2010, 0909.0801).

3. Environment Modelling: Action-Conditional Context Tree Weighting

MC-AIXI-CTW approximates the Bayesian mixture over all computable models by restricting to prediction suffix trees (PSTs) of bounded depth $x_t = (o_t, r_t)$8, employing Context Tree Weighting (CTW) with Krichevsky-Trofimov (KT) estimators. The input stream is constructed via bit-encoding of actions and percepts:

• KT Estimator: For context $x_t = (o_t, r_t)$9, with $o_t$0 zeros and $o_t$1 ones observed, the KT predictive probability is

$o_t$2

• PST Update: Actions are encoded and appended to a bit-buffer, with percept bits following. For each new percept bit, descend the context tree of depth $o_t$3, updating node counts via KT.
• CTW Mixture Weight: Each node $o_t$4 maintains a weighted probability $o_t$5, recursively defined as:

$o_t$6

At the root, $o_t$7 yields the total Bayesian mixture over all PSTs up to depth $o_t$8 with code-length penalization $o_t$9 (Veness et al., 2010, 0909.0801).

• Factored Action-Conditional CTW (FAC-CTW): For multi-bit actions/percepts, maintain separate context trees for each factor, increasing context depth for successive bits, and compose the product mixture.

4. Integration and Agent Cycle

The MC-AIXI-CTW agent integrates learning and planning as follows:

1. At time $r_t$0, select action $r_t$1 using ρ-UCT with the current FAC-CTW model as the generative environment.
2. (Optionally) Apply an outer exploration policy (e.g., $r_t$2-greedy).
3. Execute $r_t$3, observe $r_t$4.
4. Update FAC-CTW: Append action bits (no KT update), then for each percept bit apply KT update in context tree, updating $r_t$5.
5. Extend history with $r_t$6 and increment $r_t$7 (0909.0801).

This tight coupling allows for online Bayesian-sequence prediction and decision-making under unknown, partially observable, and non-Markovian environments.

5. Computational Complexity and Resource Profile

The computational cost and scalability of MC-AIXI-CTW are characterized by:

• CTW Update: $r_t$8 per new bit, independent of total cycles.
• CTW Sampling (for rollouts): $r_t$9 for each percept of $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$0 bits.
• ρ-UCT Simulation: $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$1 per trajectory to depth $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$2.
• Per-Action Cost: $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$3 for $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$4 simulations per real decision.
• Memory: Context tree occupies at most $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$5 nodes; UCT search tree up to $$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$6 nodes (Veness et al., 2010, 0909.0801).

Parallelization of simulations is direct due to the Monte Carlo nature of tree search. The approach is practical for moderate domain sizes, with rollouts performed at interactive rates on a single CPU core.

6. Empirical Evaluation

MC-AIXI-CTW was empirically validated on diverse environments, including:

Domain	Simulations	Cycles	Search/Cycle
Cheese Maze (POMDP)	500	$$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$7	$$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$8s
Tiger (POMDP)	10,000	$$a_t^* = \arg\max_{a_t} \sum_{x_t} \dots \max_{a_{t+m}} \sum_{x_{t+m}} \left[ \sum_{i=1}^m r_{t+i} \right] \cdot \xi_U(x_{1:t+m} | a_{1:t+m})$$9	$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$0s
TicTacToe	5,000	$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$1	$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$2s
Kuhn Poker	3,000	$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$3	$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$4s

In domains with known optima, the agent converged to near-optimal average reward within $$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$5–$$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$6 cycles. In Biased Rock-Paper-Scissors, MC-AIXI-CTW reached optimality by $$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$7 cycles, while competitors such as Active-LZ remained suboptimal after $$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$8 cycles. In all domains, learning was smoother and more reliable compared to U-Tree and BLHT baselines. In partially observable Pacman, performance scaled well with experience despite the absence of known optimal policy (Veness et al., 2010, 0909.0801).

7. Limitations and Future Directions

MC-AIXI-CTW's main limitations stem from its model class and planning resources:

• Model Class: Restriction to bounded-depth PSTs limits ability to capture rich or hierarchical perceptual structures (e.g., raw visual input).
• Planning Horizon: Optimal planning is limited by feasible horizon $$\xi_U(x_{1:n} | a_{1:n}) = \sum_\nu 2^{-K(\nu)} \nu(x_{1:n} | a_{1:n})$$9; planning over long-term deferred rewards may be ineffective without heuristic exploration (e.g., $\nu$0-greedy).
• Improvement Directions: Prospective work includes accommodating richer context representations (predicate-based, hashed), mixture with alternative model classes (e.g., Lempel-Ziv), rollout policy learning, continuous observation extension (quantization/Gaussian mixtures), and hardware-scale parallelism (0909.0801).

MC-AIXI-CTW remains the first agent to combine Bayesian sequence prediction (via FAC-CTW) with UCT-based planning, achieving provable sample efficiency and scalability in stationary finite-memory environments, and providing a design foundation for further advances in universally intelligent agents (Veness et al., 2010, 0909.0801).