What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty

Abstract: As artificial agents become increasingly capable, what internal structure is necessary for an agent to act competently under uncertainty? Classical results show that optimal control can be implemented using belief states or world models, but not that such representations are required. We prove quantitative "selection theorems" showing that low average-case regret on structured families of action-conditioned prediction tasks forces an agent to implement a predictive, structured internal state. Our results cover stochastic policies, partial observability, and evaluation under task distributions, without assuming optimality, determinism, or access to an explicit model. Technically, we reduce predictive modeling to binary "betting" decisions and show that regret bounds limit probability mass on suboptimal bets, enforcing the predictive distinctions needed to separate high-margin outcomes. In fully observed settings, this yields approximate recovery of the interventional transition kernel; under partial observability, it implies necessity of belief-like memory and predictive state, addressing an open question in prior world-model recovery work.

• The paper demonstrates that achieving low average-case regret in structured tasks necessitates the emergence of predictive internal representations.
• It details a reduction of predictive modeling to binary betting decisions, leading to provable bounds on transition kernel recovery and belief formation.
• The results reveal that minimized regret forces agents to adopt modular, regime-sensitive architectures with decision-relevant internal states.

Selection Theorems for Internal Structure in Robust Decision-Making under Uncertainty

Motivation and Problem Statement

The paper "What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty" (2603.02491) addresses the question: What internal representations are forced upon agents that achieve competent, robust decision-making under uncertainty? Classical results demonstrate that belief states and world models are sufficient for optimal control in POMDPs, but these results do not establish necessity. That is, optimality can be achieved via belief-like representations, but an agent may also succeed without modeling predictive structure if its tasks or evaluation regime do not demand it. The paper closes this gap by proving quantitative "selection theorems" that show low average-case regret in structured action-conditioned prediction tasks compels agents to instantiate predictive internal state.

Conceptual Framework and Technical Approach

The paper operates in the setting of sequential decision-making—both fully observed MDPs and partially observed POMDPs—with goal-conditioned evaluation protocols. It departs from previous work by not assuming worst-case optimality or determinism, and instead considers the average normalized regret achievable by stochastic policies across a family of diagnostic prediction tasks.

The central technical innovation is reducing predictive modeling requirements to families of binary "betting" decisions. Regret decomposition reveals that average-case performance bounds constrain the probability mass assigned to suboptimal betting branches. When evaluation regimes include large-margin tests (where predictive uncertainty is not trivial), agents are forced to refine their internal state to resolve the predictive distinctions necessary for effective decision-making. In fully observed environments, this yields approximate recovery of the transition kernel; in partially observed settings, it enforces belief-like memory and predictive representations, thereby formalizing when such structure is necessary rather than merely sufficient.

Key Results and Theorems

Fully Observed Environments: Transition Kernel Recovery

For MDPs where the agent observes true state and transitions, the paper constructs composite goal families that encode multi-attempt binary betting tasks. A central theorem demonstrates that, under a bound $\bar\delta$ on average case regret over diagnostic task families, a stochastic policy’s choices encode a “soft” estimator $\widehat P_{ss'}(a)$ of the transition kernel with provable mean absolute error:

$$\mathbb{E}_{(s,a,s')}\left[|\widehat P_{ss'}(a) - P_{ss'}(a)|\right] \leq \frac{t_\gamma}{\sqrt n} + \frac{\bar\delta}{c(\gamma)} + O\left(\frac{1}{n}\right)$$

where $t_\gamma, c(\gamma)$ are constants determined by the margin threshold $\gamma$. This formalizes that competency over multi-step tasks requires encoding transition structure with precision increasing in task horizon.

The causal content is also characterized: satisfaction of these regret constraints yields recovery of the interventional kernel $P(S_{t+1} \mid S_t, \mathrm{do}(A_t=a))$ (Pearl’s Level 2 interventions) to within error determined by the average regret bound and structural mismatch. However, crucially, the paper proves that recovery of Level 3 (counterfactual) queries is not generically possible from these constraints—two SCMs can yield identical interventional kernels yet differ in their counterfactual couplings.

Partial Observability: Predictive-State and Memory Necessity

In partially observed settings, the betting reduction applies to predictive-state tests (PSRs), where the agent bets on future observation sequences under prescribed action trajectories. The main theorem states that, for any margin $\gamma$, low global average-case regret forces the agent to place vanishing probability mass on suboptimal bets for large-margin tests—quantifying the degree of internal structure needed.

Moreover, the necessity of belief-like memory is proved via “no-aliasing” bounds: any memory statistic $M$ that aliases histories requiring opposite confident bets incurs unavoidable pairwise regret. Concretely, if an agent’s internal memory fails to distinguish histories which the optimal bets would separate with margin $\gamma$, then average pair-regret is forced to be at least $q^{\mathsf{Alias}_\gamma(M)} c(\gamma)/2$, where $q^{\mathsf{Alias}_\gamma(M)}$ is the mass of such aliasing events.

Structuring Internal Organization from Task Family

The paper further derives corollaries for modularity, regime sensitivity, and representational match:

• Informational modularity: Block-structured task distributions select for modular memory representations that distinguish high-margin predictive tests within each block.
• Regime tracking: Mixtures of task distributions select for persistent regime-tracking state; low regret across regimes compels memory to encode regime-sensitive distinctions.
• Representational match: For vanishing-regret agents under a common margin-minimality requirement, their internal representations must agree up to invertible recoding on decision-relevant partitions, formalizing convergence of internal structure under shared competence constraints.

Practical and Theoretical Implications

These selection theorems establish that average-case regret bounds over structured predictive tasks impose quantitative, representation-theoretical constraints on agent architecture. The results connect capability with necessity: robust competence in uncertain, multi-regime environments selects for world modeling, belief-like memory, modularity, and persistent internal variables—not as optional architectural features, but as consequences of empirical task demands.

This provides a rigorous underpinning for the emergence of convergent internal representations in deep RL agents, world model architectures, and even biological neural systems. The theory aligns with empirical findings across visual, auditory, motor, and language modalities, as well as hypotheses in NeuroAI and cognitive science regarding the origin of modularity, global workspace, and unified predictive representations. The selection framework offers a principled explanation for observed representational alignments across artificial and biological systems, supporting modalities such as whole-brain data in animal studies evaluated against agent models.

From an engineering perspective, as AI systems become increasingly general and robust, the results suggest organizational signatures—predictive state, regime tracking, and modular decomposition—will emerge inevitably. This supports efforts in interpretability, capability benchmarking, and possibly AI alignment, as internal representations can be analyzed for decisional sufficiency and structural integrity.

Speculation on Future Directions

Future research may extend these selection theorems to richer task distributions, continuous domains, multi-agent settings, and hierarchical memory models. There is potential to leverage the theory both for designing architectures that systematically enforce desired internal structures and for empirically testing representational properties of advanced agents and biological systems.

Further, the explicit distinction between representation necessity and recovery suggests new avenues for understanding and diagnosing internal states in agents based on regret-based behavioral signatures. These results may inform philosophical and empirical approaches to emergent consciousness and agency in artificial systems, by formalizing which aspects of internal state become structurally mandatory under competent, uncertainty-sensitive operation.

Conclusion

The paper rigorously formalizes how robust, regret-bounded generalization under uncertainty compresses the admissible space of internal agent architectures, compelling belief-like predictive modeling, decision-sufficient memory, modular decomposition, and regime-sensitive internal variables. It separates the sufficiency and necessity of world modeling and establishes tractable quantitative constraints for both fully observed and partially observed settings. Selection theorems connect empirical competence to structural inevitability, offering a principled foundation for interpreting the emergence of internal representation and organization in increasingly capable artificial agents (2603.02491).