This paper, authored by Cole Wyeth and Marcus Hutter, explores the limitations inherent in the AIXI reinforcement learning agent as a representation of embedded agency within the framework of universal artificial intelligence (UAI). AIXI has been historically perceived as a formal gold standard for AGI; however, this research critically examines its embeddedness failures owing to its Cartesian dualist nature. In this context, the agent interacts with an environment externally but does not possess self-awareness of its existence within that environment. Consequently, the paper investigates modifications to the AIXI agent that attempt to account for embedded agency.
Central to the investigation is the introduction of a variant called joint AIXI, characterized by modeling the joint action/percept history using the universal distribution. This modeling is meant to integrate the agent's actions as explainable by the same principles governing percept generation, thereby attempting embeddedness. While the paper clarifies this method as intuitive, it brings attention to its lack of detailed analysis alongside frequent assumptions (not proofs) of its failure.
Key Contributions and Numerical Results
The report systematically details the mathematical preliminaries underlying UAI and explores the notions of algorithmic information theory needed to comprehend sequences and environment distributions. It further discusses sophisticated approaches to embedded AIXI variants proposed by other researchers, among which reflective oracles play a significant role, providing computable solutions at parity with the environments within the hypothesis class.
Reflective AIXI utilizes these oracles to transform perceived interactions into actionable conditional computations, sidestepping several realizability constraints. This approach is cited as offering improved approximation computability compared to AIXI, which can be notoriously hard without s-approximation. Despite enhancing the agent’s embeddedness concerns, reflective AIXI remains computationally unbounded, presenting questions regarding recursive self-improvement.
Joint AIXI and its implications are thoroughly analyzed. Adversarial learning results indicate that while Solomonoff normalization allows deterministic environment learning, joint AIXI encounters non-convergence specifically when action bits are adversarially selected. The paper provides formalized proofs demonstrating the agent's failure to converge in simple scenarios owing to these factors.
Theoretical Impacts and Future Directions
The research enriches theoretical understanding by elucidating the complexities embedded agents must overcome, shedding light on why joint AIXI was presumed to face difficulties. Reflective AIXI models afford a promising direction, yet computational challenges predominate, leaving room for ongoing inquiry. Potential exists for refining agent designs to ensure reliable convergence amidst stochastic environments, expanding the scope of UAI effectively utilizable for AGI systems.
Conclusion
The paper concludes with detailed results justifying Marcus Hutter's proposal of a universal mixture for chronological semimeasures as a foundation for Cartesian agents. It opens several paths for future research, particularly concerning solutions that balance between the adversarial cases handled by joint AIXI and effective policy determination across reasonable environments. It marks a pivotal step toward resolving practical and theoretical dilemmas inherent in embedded AI systems, while acknowledging open questions that require deeper investigation. The findings herein underscore the enduring challenge of synthesizing UAI with an embedded agent's reality, laying groundwork for subsequent advancements in AGI frameworks.
