Prospective Learning & Embedded Agency

• Prospective learning is an approach where agents anticipate future outcomes by balancing goal-directed actions with information-seeking behavior.
• Embedded agency emphasizes designing agents that are integral to their environment by incorporating body, sensor, and world dynamics into internal generative models.
• Recent frameworks integrate Bayesian cognition, active inference, and cycle consistency in RL to enable robust, forward-looking decision-making.

Prospective learning refers to the ability of an agent to anticipate and evaluate the future consequences of its actions, integrating both instrumental (goal-directed) and epistemic (information-seeking) considerations over extended horizons. Embedded agency, on the other hand, concerns the modeling and design of agents that are not mere external optimizers but are inseparably situated within the environments they act upon, with their body, sensors, and world-dynamics forming a closed system. In recent years, research has converged on formal frameworks that reconcile prospective learning and embeddedness—chiefly active inference, agency-preserving AI, and prospective RL—grounding agency in Bayesian cognition, control theory, and mechanistic learning paradigms.

1. Generative Models, Perception, and Embeddedness

Agents operating under embedded agency theories posit a generative model, typically over sequences of latent states $s_{1:T}$ and observations $o_{1:T}$. For an active-inference agent, the model is expressed as

$$P(o_{1:T}, s_{1:T}) = P(s_1) \prod_{t=1}^T P(o_t \mid s_t) P(s_t \mid s_{t-1}, a_t)$$

where $s_t$ and $o_t$ may encode both external and internal (bodily) variables. Importantly, $P(s_t \mid s_{t-1}, a_t)$ embodies body–world coupling, such as Newtonian or musculotendon dynamics, and $P(o_t \mid s_t)$ encapsulates sensor mechanisms (e.g., retina, skin, cochlea). This formalism ensures that agent-environment coupling is expressed within the generative model itself, without ad-hoc modularity, thus satisfying the embeddedness criterion.

For Bayesian belief updating, agents introduce a recognition model $Q(s_{1:T} \mid o_{1:T})$, typically factorized for tractability. Minimization of the variational free energy

$$F[Q(\cdot \mid o_{1:T})] = \mathbb{E}_{Q(s_{1:T} \mid o_{1:T})}\left[\ln Q(s_{1:T} \mid o_{1:T}) - \ln P(o_{1:T}, s_{1:T})\right]$$

serves as the principle for state inference ("perception") (Costa et al., 23 Jan 2024).

2. Expected Free Energy and Prospective Learning

Planning under active inference is formalized through the minimization of expected free energy $G(\pi)$ for policies $\pi$. This functional is given by

$$G(\pi) = \mathbb{E}_{Q(o_{t+1:T}, s_{t+1:T} \mid \pi)} \left[ \ln Q(s_{t+1:T} \mid \pi) - \ln P(o_{t+1:T}, s_{t+1:T}) \right]$$

which, upon decomposition, yields risk (extrinsic, preference-aligned) and ambiguity (intrinsic, information-seeking) terms:

• Risk: $D_{\mathrm{KL}}[P(s_\tau \mid \pi) \| P(s_\tau)]$
• Ambiguity: $$\mathbb{E}_{P(s_\tau \mid \pi)}[H[P(o_\tau \mid s_\tau)]]$$

Thus, minimization of $G(\pi)$ enforces a rational trade-off between exploitation (preference satisfaction) and exploration (epistemic gain), with all such differences rendered explicit in the generative model’s structure (Costa et al., 23 Jan 2024).

This policy selection mechanism

$Q(\pi) \propto \exp(-G(\pi))$

ensures that the agent prospectively simulates and assigns weight to futures based on both desirability and epistemic leverage—the defining criterion of prospective learning.

3. Agency Preservation, Forward-Looking Functionals, and Embedded Learners

Agency-preserving AI, as expounded by Mitelut et al., postulates a distinct, forward-looking evaluation of agency in human-AI interaction. The central formalism introduces an agency functional for a possibly AI-influenced human policy $\pi_H$:

$$A_H(\pi_H, s_0) = \mathbb{E}_{\tau \sim \pi_H}\left[ \sum_{t=0}^\infty \gamma^t r_A(s_t, a_t) \mid s_0 \right]$$

where $r_A(s, a)$ rewards the preservation of future options—the agent's degrees of freedom—distinct from task-centric reward signals. The objective is to ensure

$$A_H(\pi_H^{AI}(\theta), s_0) \geq A_H(\pi_H^{\text{baseline}}, s_0) - \epsilon$$

and to define agency loss accordingly. Illustrative simulations in temporal-difference environments demonstrate that “intent alignment” at each step is not sufficient to preserve forward-looking agency: short-term nudges can corral embedded learners into low-agency regions of the state space, with measurable reduction (up to 40%) of $A_H$, as observed in chain MDP and bandit settings (Mitelut et al., 2023).

Thus, the requirement for AI systems to employ prospective learning—not merely retrospective imitation—directly addresses the agency risk posed by embeddedness and nontrivial environment dynamics.

4. Prospective RL: Internal World Models and Cycle Consistency

ProSpec RL introduces a concrete instantiation of prospective learning by bridging model-free behavior with internalized, model-based “imagination.” The agent constructs a Flow-based Dynamics Model (FDM) $h_\theta$ to encode transition dynamics in latent space:

• Observation encoding: $z_t = f(s_t)$
• Transition: $\hat z_{t+1} = h_\theta(a_t, z_t)$

Agents sample $k$ candidate action sequences $\hat{A}^{(i)}$, propagate them forward via $h_\theta$, and estimate their prospective (“prospected”) return using a learned critic $Q_\phi(z,a)$. The action maximizing expected return among $k \times H$ plans is selected. Planning is thus performed via one-step Model Predictive Control (MPC) with online re-planning.

An innovative cycle consistency constraint is enforced: $h_\theta^{-1}$ (analytic, owing to RealNVP) must be able to reconstruct $z_t$ after random multi-step forward-then-backward transitions. The cycle-consistency loss

$$\mathcal{L}_c(\theta) = \frac{1}{M} \sum_{m=1}^M d(z_t^{(m)}, \breve z_t^{(m)})$$

penalizes irreversible, risk-prone imagined futures, thereby augmenting agent caution and data efficiency (Liu et al., 31 Jul 2024).

Empirical validation on DMControl tasks shows significant performance improvements over baselines, with ablations confirming the necessity of prospective rollout and cycle consistency for optimal behavior.

5. Core Commitments and Comparison to Standard RL

Active inference and ProSpec RL, while distinct in mechanics, both realize embedded agency through internal world modeling and prospective evaluation:

• The agent’s generative model encodes full body–environment couplings; policies are evaluated by simulating their long-term effects within this internalized model structure.
• There is no need for hand-crafted exploration bonuses: exploration derives directly from the epistemic components of the planning objectives ($G(\pi)$ in active inference; cycle consistency in ProSpec RL).
• Value functions are not primary objects: in active inference, action selection follows from free-energy ($G(\pi)$) or its surrogates, rather than value iteration (Costa et al., 23 Jan 2024).
• Standard RL approaches implement intent alignment by maximizing cumulative reward, often with myopic or extrinsic heuristics. Embedded agency frameworks instead ground action selection in prospective assessment of both future option sets and epistemic gains, tracking agency loss and option richness (Mitelut et al., 2023).

6. Toward Agency Foundations: Open Problems and Research Directions

There is an emerging research agenda targeting robust agency preservation and prospective learning in the context of embedded agency. Four initial research topics are critical (Mitelut et al., 2023):

• Benevolent Game Theory: Solution concepts ensuring agency functionals are non-decreasing under rational play.
• Algorithmic Foundations of Human Rights: Encoding rights as constraints on long-horizon agency functionals and developing enforcement algorithms.
• Mechanistic Interpretability: Analyzing internal representations of options and degrees of freedom in neural policies.
• Reinforcement Learning from Internal States: Developing agents that infer and maximize latent preferences over choice-rich futures, transcending superficially observed actions.

Table: Formalisms Linking Prospective Learning and Embedded Agency

Framework	Prospective Mechanism	Embeddedness Realization
Active Inference	$G(\pi)$ minimization	Explicit generative body-world model
Agency-Preserving AI	$A_H(\pi_H, s_0)$ evaluation	Forward-looking agent-environment-state dependency
ProSpec RL	Imagined rollouts & cycle consistency	Latent world model, reversible planning

This suggests a convergence across disciplines—Bayesian cognition, RL, control theory—on the necessity of endowing artificial agents with internal models, forward-looking evaluation functionals, and embedded-world reasoning to accomplish robust agency preservation, safe intent alignment, and adaptive, risk-sensitive planning.