Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness

Abstract: Classical reinforcement learning assumes the agent interacts with a fixed environment whose behavior does not depend on the agent's policy. This assumption breaks down in non-realizable settings where other actors might anticipate the agent's behavior, including environments crucial to AI safety, where the agent interacts with predictors, humans, other AI agents, and institutions. In such settings, the agent's model class fails to capture the world in which it operates. Under such misspecification, classical Bayesian methods can produce confidently wrong posteriors, unreliable decisions, and unbounded regret, as realizability fails to obtain. Infra-Bayesianism is a decision-theoretic framework that addresses these failures by distinguishing ordinary probabilistic uncertainty, where priors can be reasonably chosen, from Knightian uncertainty, where no grounds exist for the construction of such a prior. It does so by evaluating actions on their worst-case outcomes, rather than from posterior expectations or weighted averaging. We present the first proof-of-concept implementation of an infra-Bayesian reinforcement learning architecture for finite-outcome stateless decision problems. Our agent maintains a set of imprecise hypotheses, updates them using infra-Bayesian conditioning, and selects actions by maximizing worst-case expected value. We apply this implementation of the infra-Bayesian maximin decision process to an environment with Knightian uncertainty, and demonstrate a lower worst-case regret as compared to classical reinforcement learning agents. We also investigate Newcomb's problem and show that the infra-Bayesian agent picks the optimal strategy, outperforming classical decision theory agents. Our results provide a step towards reinforcement learning agents that remain robust under model misspecification and policy-dependent uncertainty.

• The paper presents a formal framework and implementation of infra-Bayesian RL that leverages affine measures to account for Knightian uncertainty.
• Empirical results demonstrate that IB agents achieve lower worst-case regret and outperform classical Bayesian methods in adversarial bandit and policy-dependent scenarios.
• The work implies robust agent design for high-stakes applications, though it may sacrifice value in benign settings due to its cautious, worst-case planning strategy.

Infra-Bayesian Reinforcement Learning: Formalism and Implementation

The paper "Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness" (2605.23146) presents a formal, empirical, and computational study of infra-Bayesian (IB) decision theory in reinforcement learning (RL). By explicitly constructing and evaluating IB RL agents, the authors bridge the theoretical framework of infra-Bayesianism—originally developed to address misspecification and Knightian uncertainty in embedded agency—with tractable agent implementations. This essay provides a comprehensive analysis of the paper's contributions, results, implications, and future directions.

Motivation: Limitations of the Classical Bayesian Paradigm

Classical RL and Bayesian RL make a realizability assumption: the agent's hypothesis class contains (with high probability) the true environment. This enables the use of precise priors and posteriors, supporting convergence and regret guarantees. However, in real-world AI deployment, especially in open-ended or policy-dependent environments—where other actors may anticipate and react to the agent's policy—such assumptions fail. Model misspecification becomes inevitable, and Bayesian agents can become confidently wrong, yielding unreliable posteriors and unbounded regret. IB extends beyond this by formalizing reasoning in the face of both ordinary (probabilistic) and Knightian (ambiguous, not classically reducible) uncertainty.

The Infra-Bayesian Formalism: Affine Measures and Infradistributions

Infra-Bayesianism replaces single probability distributions with sets of admissible evaluators, termed a-measures, paired with a set-based construct called an "infradistribution". An a-measure $a=(\lambda\mu,b)$ combines a base probabilistic measure $\mu$ (weighted by $\lambda$) and an offset $b$ capturing value ruled out by observations—ensuring dynamic consistency. Policies are evaluated not by their Bayesian expected value, but by the lower expectation over the entire infradistribution:

$$\underline{\mathbb{E}_{\Psi}[f]} = \inf_{a\in\Psi} a(f)$$

This formalizes a worst-case, maximin criterion over possible world models, with explicit representation of both classical mixtures (ordinary probabilistic uncertainty) and Knightian mixtures (ambiguity/uncertainty that cannot be probabilistically averaged).

Design and Computational Architecture

The implementation targets finite-outcome, stateless RL problems (notably bandits and Newcomb-like decision problems). Representations focus on:

• Infradistributions as Convex Polytopes: Only extremal (vertex) minimal points need to be stored for correctly computing lower expectations, analogous to credal set convex hulls.
• World Models: Specialized for Bernoulli bandits and Newcomb-like environments, factoring histories and measures to ensure computational tractability.
• Decision and Update Procedures: At each timestep, all candidate policies $\pi$ are evaluated via $\underline{\mathbb{E}_{\Psi(\pi)}[f]}$; the action/policy maximizing the lower expectation is selected. Updates use the IB conditioning rule, adjusting the affine parameters according to both realized and ruled-out histories.

Empirical Results: Validation and Robustness

Classical Limit

The agent correctly reproduces classical Bayesian behavior for appropriate (singleton) initialization, serving as a validity check for the formalism and the codebase (Figure 1).

Figure 1: Validation that the infra-Bayesian agent reproduces classical Bayesian bandit behavior in the precise-prior, single-$a$-measure case. The legend gives the priors of each arm in four unique Bernoulli bandit settings.

Robustness Under Knightian Uncertainty

A salient demonstration is provided in two-armed, adversarial Bernoulli bandit settings with interval constraints on arm probabilities, but no precise probabilistic model. The IB agent, representing true Knightian uncertainty, outputs a robust deterministic policy maximizing worst-case return; classical Bayesian agents are compelled to commit to specific priors, yielding potentially suboptimal policies and higher worst-case regret.

Figure 2: Comparing the performance of infra-Bayesian and classical Bayesian agents (with either greedy or Thompson Sampling exploration strategies) in the trap bandit setting. The first row shows results for a correctly specified Bayes prior condition; the second for a severely misspecified Bayes prior condition. The first column shows cumulative expected regret, and the second shows the average pull rate of the risky (trap) arm.

Numerical results clarify that the IB agent realizes consistently lower worst-case regret over possible bandit configurations. The ability to avoid catastrophic outcomes, even when learning is infeasible (i.e., rewards are selected adversarially each round), emerges directly from the formal representation of Knightian uncertainty.

Performance in Newcomb-like and Policy-Dependent Environments

The IB architecture naturally accommodates policy-dependent environments, where policies—not just actions—influence transitions/rewards. In Newcomb's problem (with imperfect predictors), the IB agent consistently recovers the optimal strategy: one-boxing when the predictor's accuracy exceeds a regime-dependent threshold and two-boxing otherwise. Classical decision theories (causal, evidential) either fail to maximize reward or do so for the wrong causal reason, failing to reflect the problem structure. Simulations confirm IB planning achieves upper-bound optimality in all tested regimes.

Theoretical and Practical Implications

The empirical findings substantiate longstanding theoretical claims: IB RL agents can offer stronger worst-case guarantees—of value and regret—in environments featuring model misspecification, ambiguous/Knightian uncertainty, and policy dependence, where Bayesian RL is ill-posed or brittle. IB agents do not require an arbitrary, potentially misleading prior over world models and are not forced to update toward confidently inaccurate conclusions when the true world lies outside their representational class.

Practically, such robustness is critical in high-stakes, open-world, and AI safety–relevant domains, where catastrophic divergence or spurious confidence in misaligned posteriors is unacceptable.

Limitations and Directions for Future Work

The current implementation demonstrates the IB paradigm only for finite, stateless problems, leveraging full enumeration of extremal points for tractability. Extensions to:

• Sequential/temporal (multi-step) problems with history dependence
• Continuous or large state/action spaces requiring function approximation
• Computational scaling and algorithmic efficiency in high-dimensional infradistribution representations

are clear priorities for subsequent research.

Additionally, the cost of robustness under Knightian uncertainty is systematically observed: IB agents may forgo value in benign settings (pessimistic policies) to secure catastrophic risk bounds. Quantifying and managing this tradeoff against Bayesian methods, including adaptive transitions between IB and Bayesian representations, represents a fertile area for further theoretical and empirical study.

Conclusion

This work delivers a formal, empirically validated architecture for infra-Bayesian reinforcement learning. It substantiates that IB agents—by explicitly distinguishing probabilistic and Knightian uncertainties, and planning against worst-case lower expectations—can achieve lower worst-case regret and robust optimality in misspecified or policy-dependent environments, a property classical Bayesian RL demonstrably lacks. These results provide early, concrete steps toward theoretically justified, robust agent architectures suitable for high-assurance autonomous AI deployment.