Self-Predictive Agents: The Self-AIXI Model

• Self-AIXI is a reinforcement learning agent that integrates universal Bayesian prediction, planning, and self-modeling to achieve near-optimal decision making in computable environments.
• It incorporates intrinsic motivations such as variational empowerment and free energy minimization to drive exploration and adaptive behavior in uncertain, complex systems.
• Approximation methods like MC-AIXI(CTW) and DynamicHedgeAIXI translate the idealized Self-AIXI framework into practical algorithms that emphasize self-improvement and safety.

Self-predictive agents, operationalized in the literature as Self-AIXI, are reinforcement learning systems that recursively integrate prediction, planning, and self-modeling under a universal Bayesian formalism. Originating from the union of sequential decision theory and algorithmic probability, Self-AIXI extends the theoretical AIXI agent by embedding intrinsic objectives such as empowerment and metacognitive self-assessment, supporting robust adaptation, exploration, and self-improvement even in highly uncertain, complex environments. These agents aim for Bayes-optimality with respect to all computable environments, but unlike static AIXI models, Self-AIXI also predicts and evaluates its own future actions and learning process, enabling capabilities such as self-modification, value learning, and adaptive response to epistemic uncertainty.

1. Fundamentals of Self-AIXI and Universal Prediction

Self-AIXI is grounded in the principles of algorithmic probability, as realized in Solomonoff induction and extended to interactive RL in AIXI (Sunehag et al., 2011). Key mechanisms include:

• Universal computability: All predictions about the external environment and the agent’s own behavior are computed as Bayesian mixtures over the space of all computable models, each weighted by program length to enforce Ockham's razor.
• Rationality: The policy is selected to maximize Bayesian expected reward, conditioning not only on observations but also on the agent's own hypothesized future decisions.
• Indifference and time consistency: Priors are set symmetrically over the hypothesis space and updated via strict Bayesian conditioning, ensuring stability in sequential decision making and over self-referential prediction.

Formally, the idealized agent at each timestep selects actions maximizing

$$a^*_{t+1} = \arg\max_a \sum_{x_{t+1}} \ldots \max_{a_{t+m}} \sum_{x_{t+m}} \left[\sum_{i=t+1}^{t+m} r_i\right] \sum_{\rho\in\mathcal{M}}2^{-K(\rho)}\rho(x_{1:t+m}|a_{1:t+m}),$$

where $K(\rho)$ is the Kolmogorov complexity of the environment hypothesis $\rho$ (Veness et al., 2010).

2. Intrinsic Motivation and Empowerment in Self-Predictive Agents

Recent work demonstrates that Self-AIXI agents naturally implement variational empowerment as an intrinsic exploratory drive (Hayashi et al., 20 Feb 2025). Empowerment is formally the maximal mutual information between action sequences and future states: $$\mathcal{I}(z_k; h_{<t+k}) = \max_p I(z_k; h_{t:t+k} \mid h_{<t}).$$ Self-AIXI’s decision rule includes a term

$$\sum_{i=0}^{k-1} D_{\mathrm{KL}}(\pi^*_i \Vert \zeta_i)$$

for the KL divergence between optimal and current policies, which acts as a regularizer and can be interpreted as a variational empowerment bonus. Planning is recast as minimizing expected variational free energy: $$\mathcal{F}_\varphi(z_k; h_{<t+k}) = D_{\mathrm{KL}}(p(z_k, h_{t:t+k} | h_{<t}) \| q_\varphi(z_k, h_{t:t+k}| h_{<t})),$$ where the agent seeks actions that jointly maximize external reward and intrinsic measures of control and curiosity. This dual incentive systematically yields power-seeking behavior—i.e., seeking high-optionality states—even absent explicit extrinsic reward (Hayashi et al., 20 Feb 2025).

3. From Theoretical Ideal to Computational Implementation

While AIXI and Self-AIXI provide gold standards for general intelligence, their direct computation is incomputable (Leike et al., 2015). Approximations like MC-AIXI(CTW) and DynamicHedgeAIXI address this bottleneck:

Approximation	Environment Model Class	Planning Mechanism
MC-AIXI(CTW)	Bayesian mixture over PSTs	Monte Carlo Tree Search
DynamicHedgeAIXI	Dynamically injected predicate models	Hedge-based Bayesian mixture

• MC-AIXI(CTW): Uses action-conditional Context Tree Weighting to construct a Bayesian mixture and a Monte Carlo tree search algorithm for planning. Empirically, this approach produces self-predictive behavior, robust adaptation, and near-optimal performance in benchmark POMDPs (Veness et al., 2010).
• DynamicHedgeAIXI: Allows online human-in-the-loop injection of new models with a time-adaptive prior, enabling the agent to overcome model bias and rapidly shift its predictions as superior hypotheses are supplied. The Hedge-based prior update ensures that the agent's mixture approximates the performance of the best available model (Yang-Zhao et al., 2023).

The theoretical foundation of these approximations is a shift to limit-computable or $\epsilon$-optimal value estimation with recursive definitions, retaining convergence guarantees (Leike et al., 2015).

4. Self-Prediction, Metacognition, and Value Alignment

Self-predictive agents incorporate metacognitive components:

• Metacognitive knowledge encodes self-assessment of competencies and learning strategies.
• Metacognitive planning enables self-directed selection of learning tasks and resource allocation.
• Metacognitive evaluation provides a closed-loop mechanism for updating self-knowledge and modifying learning policies in response to observed outcomes.

This triad is formalized as a bi-level optimization: $\pi^* = \arg\max_\pi \mathbb{E}[R(\pi(K, t))]$ where $K$ denotes the agent’s evolving self-knowledge vector and $t$ designates task. Regular updates ensure continual improvement and robust adaptation, while enabling the agent to autonomously decide when to seek human guidance or alter its long-term learning trajectory (Liu et al., 5 Jun 2025).

Furthermore, hierarchical value learning and ethical bias frameworks propose that, instead of maximizing external rewards directly, agents should infer values compatible with human or mature agent goals from social context, using internalized hierarchical representations and minimum description length principles. This supports safer, more aligned self-improvement trajectories in multi-agent environments (Potapov et al., 2013).

5. Practical Implications: Exploration, Safety, and Controllability

Self-predictive agents offer several technical advantages:

• Power-seeking and safety: The intrinsic drive for empowerment, while essential for exploration and competence, also presents AI safety challenges, as agents naturally gravitate toward high-control states. This duality requires the design of safe exploration and alignment mechanisms accounting for both external rewards and intrinsic empowerment signals (Hayashi et al., 20 Feb 2025).
• Episodic and boxed operation: Variants such as BoMAI restrict the agent’s influence to controlled environments and employ information-theoretic exploration schedules, preventing reward hijacking and undesirable instrumental strategies (Cohen et al., 2021).
• Sense of self and ethical reasoning: Elastic self models expand the agent’s internal valuation to a weighted set of identities, supporting the emergence of cooperative and ethically conscious behavior, as the agent internalizes broader societal norms within its utility function (Srinivasa et al., 2022).

6. Advances in Representation Learning and Empirical Performance

Recent developments emphasize the role of action-conditional self-predictive representation learning:

• BYOL-AC and variance objectives: Action-conditional self-predictive objectives result in latent spaces that optimally encode per-action dynamics, supporting improved decision quality. Theoretical ODE analyses guarantee convergence to subspaces maximizing the spectral variance across transitions, with empirical superiority in both linear and deep RL domains (Khetarpal et al., 4 Jun 2024).
• Self-supervised temporal prediction: SPR (Self-Predictive Representations) architectures combine self-supervised future prediction in latent space with data augmentation, yielding substantial sample efficiency improvements (median human-normalized score 0.415 on Atari 100k benchmarks) (Schwarzer et al., 2020).
• Transferable predictive modeling: Adaptive modularization of goals and predictors enables agents to transfer predictive models to new tasks, with rapid adjustment accomplished via evolving goal networks, facilitating robust performance in novel or adversarial scenarios (Ellefsen et al., 2019).

7. Outlook and Open Research Directions

Self-AIXI systems consolidate the principle of universal induction with recursive self-assessment and empowerment maximization. Key challenges include scalable computable approximations, principled value alignment, and robustly partitioning metacognitive responsibilities between agents and human overseers. Regulation of intrinsic motivational signals (empowerment, curiosity) is necessary to preempt unintended power-seeking. Advances in model adaptation (e.g., dynamic knowledge injection) and self-supervised latent representation learning provide critical technical leverage. Continued research is required to both ground safety assurances and realize general, self-improving artificial agents that reconcile autonomy with value-aligned behavior in nontrivial, structured environments.