---
title: Free Energy Projective Simulation
url: https://www.emergentmind.com/topics/free-energy-projective-simulation-feps
type: topic
---

# Free Energy Projective Simulation

Free Energy Projective Simulation (FEPS) is a framework that models agents as interpretable, graph-based systems performing active inference in partially observable environments. FEPS integrates the free energy principle (FEP) and active inference (AIF) with a structured memory architecture, enabling agents to derive optimal policies via internal reward mechanisms and explicit world models without relying on external scalar rewards or deep neural networks [2411.14991].

## 1. World Model and Internal Representation

FEPS agents maintain an explicit world model rooted in a partially observable Markov decision process (POMDP). The model components are as follows:

- **Belief-state space**: $B = \{b\}$ (with $b$ denoting "clone clips").
- **Observation space**: $S = \{s\}$ (sensory states).
- **Action space**: $A = \{a\}$.
- **Transition function**: $T$, representing $p(b'|b, a)$, models transitions between belief states given actions.
- **Emission (likelihood) function**: $L$, representing $p(s|b)$, associates belief states with observations.
- **Internal reward**: $R$, used for learning (see Section 3).

A key architectural feature is the clone-structured Episodic & Compositional Memory (ECM). Each observation $s\in S$ is associated with $N_\text{clone}$ "clone clips" $b\in B$, yielding $|B| = N_\text{clone} \cdot |S|$. The emission edges $b \to s$ are deterministic ($p(s|b)=\delta_{s,s(b)}$). Transition edges $b \xrightarrow{a} b'$ carry trainable weights $h_{b,b'}^{(a)}$ encoding $p(b'|b,a)$. Deliberation in FEPS corresponds to a random walk on the ECM, while policy selection is implemented via a bipartite graph from $B$ to $A$.

The full joint distribution across time $t$ is:

$$
p(B_{0:t}, A_{0:t-1}, S_{0:t}) = p(B_0, S_0) \prod_{\tau=1}^t \pi(A_{\tau-1}|B_{\tau-1})\,p(S_\tau|B_\tau)\,p(B_\tau|B_{\tau-1}, A_{\tau-1})
$$

## 2. Expected Free Energy and Policy Construction

Policy optimization in FEPS is regulated by minimizing expected free energy (EFE), in line with AIF:

- For current belief $b_t$ and candidate action $a$, the one-step predictive model is $p(B_{t+1}, S_{t+1}|b_t,a) = p(b_{t+1}|b_t,a)p(s_{t+1}|b_{t+1})$.
- The expected free energy for action $a$ is

$$
G_{b_t}[a] = \mathbb{E}_{b',s' \sim p(\cdot|b_t,a)} \left[\log p(b'|b_t,a) - \log \operatorname{pref}(s',b'|b_t,a)\right] = -H[B_{t+1}|b_t,a] + \mathbb{E}_{b',s'}[S^\text{pref}(s',b'|b_t,a)]
$$

with $S^\text{pref}(s',b'|\cdots) = -\log \operatorname{pref}(s',b'|\cdots)$ and $H$ the conditional entropy.

EFE decomposes as:

- **Epistemic value**: Expected information gain, corresponding to entropy reduction.
- **Pragmatic value**: Expected utility for matching the preference distribution $\operatorname{pref}$.

The policy is determined by a softmax over the negative EFE:

$$
\pi(a|b_t) = \mathrm{softmax}(-\zeta G_{b_t}[a])
$$

where $\zeta < 0$ promotes exploitation (EFE minimization), and $\zeta > 0$ can promote exploration.

## 3. Internal Rewards and Learning Dynamics

Standard RL paradigms rely on external scalar rewards $r_t$ provided by the environment. In contrast, FEPS agents use solely internal rewards driven by prediction accuracy:

- Each transition edge $b \xrightarrow{a} b'$ is endowed with a confidence value $f_{b\to b'}$.
- During prediction, as long as the predicted observation $\hat{s}_{t+1}$ matches the actual $s_{t+1}^{\text{env}}$, confidence $f$ is incremented along the trajectory.
- Upon the first prediction error, reinforcement $R \cdot f$ is distributed to each implicated transition’s $h$-value:

$$
h^{\text{new}}_{b,b'} = h^{\text{old}}_{b,b'} - \gamma(h^{\text{old}}_{b,b'} - h^0_{b,b'}) + R \cdot f_{b\to b'}
$$

with $\gamma$ the forgetting rate and $h^0$ the baseline initialization for each edge.

## 4. FEPS Operational Algorithm

FEPS proceeds through episodic interaction and online model adaptation. The essential workflow is summarized as:

```python
# Initialize ECM structure and parameters
for each observation s:
    create N_clone clone-clips b with emission edge b→s
initialize all transition h-values h^0 and confidence f=0
initialize preference distribution pref

# Loop over episodes
while not terminated:
    reset environment; receive s_env
    excite all b with s(b) = s_env  # initialize candidate set C

    while episode not finished:
        # Superposed belief state estimation
        maintain candidate set C ⊆ B consistent with s_env

        # Compute EFE and sample policy
        for each b in C:
            compute G_b[a] for all a
            π(a|b) = softmax(-ζ G_b[a])
        mix π(a|b) across C to select a_t

        # Model-based prediction and update
        for each b in C:
            sample b′ ~ p_prior(b′|b, a_t)
            predict ŝ = b′→s
        take action a_t; observe new s_env

        # Confidence update
        if ŝ == s_env:
            f(b→b′)++ for all edges in trajectory
            C ← matching clones
        else:
            for each edge (b→b′) with f:
                h ← h - γ(h-h^0) + R·f
            reset f(b→b′)=0; reinitialize C to all clones for s_env

        # Policy update
        for each b:
            h_{b→a} = G_b[a]
            π(a|b) = softmax(-ζ G_b[a])
```

This workflow aligns mechanism design, learning, and planning via EFE minimization using only internal signals.

## 5. Techniques for Interpretability and Robustness

Several explicit strategies enable FEPS interpretability and facilitate robust credit assignment in the presence of partial observability:

- **Clone-Structured Representation**: Each clone-clip inherits the semantics of its source observation, and over training, clones of the same $s$ diverge, encoding distinct hidden-state contexts.
- **Belief-State Superposition**: Candidate set $C$ (clones compatible with current $s_{\text{env}}$) models superposed beliefs; prediction-consistent narrowing localizes the agent's true latent state.
- **Long-Term Goals and Look-Ahead Preferences**: Preference $\operatorname{pref}(s',b'|b)$ is factorized as $\operatorname{pref}(s') \operatorname{pref}(b'|b)$, where $\operatorname{pref}(s')$ targets specific observations and $\operatorname{pref}(b'|b)$ is dynamically propagated via a look-ahead, akin to dynamic programming:

  $$
  v_0(b) = \operatorname{pref}(s(b)),\qquad
  v_n(b') = \max \{ v_{n-1}(b'') \cdot \beta^{n-1} \cdot p(b''|b') \cdot \pi(a|b') \}
  $$

- **Reducing Prediction Errors**: Candidate set narrowing rapidly recovers the true belief, while confidence-based reinforcement selectively solidifies reliable transitions.

## 6. Empirical Validation in Behavioral Paradigms

FEPS was validated in two behavioral-biology–inspired RL settings:

**A. Timed-Response Task (Skinner Box analog):**
- Environment: Hidden MDP featuring ambiguous observations (e.g., "light on, hungry" from two different latent states).
- Metrics: Error-free trajectory length, variational free energy (VFE), and EFE evolution.
- Results:
  - Agents differentiate clones for ambiguous contexts.
  - VFE undergoes sharp drops corresponding to elimination of impossible transitions and correct context separation.
  - Look-ahead preferences enable the acquisition of correct multi-step policies.

**B. Partially Observable Grid Navigation:**
- Environment: $3 \times 3$ grid, hidden food goal, overlapping scent observations; agent actions: up/down/left/right; 3 clones per scent.
- Metrics: Trajectory length, VFE, EFE, policy optimality, and median steps to goal.
- Results:
  - For $\zeta = -3$ (task) and $\zeta = +1$ (wandering), longest trajectories achieved in most trials.
  - Superposed belief tracking maximizes error-free exploration.
  - Post-training, clone-to-cell mapping recapitulates a cognitive map.
  - Preference reconfiguration reuses the trained world model for new goals.

## 7. Limitations and Prospects for Extension

Noted constraints and open areas for FEPS include:

- **Task specification** is limited to sensory observations; direct hidden state preferences cannot be represented.
- **Scalability**: Application to large state/action spaces may require hierarchical or function-approximate augmentations.
- **Exploration-exploitation**: The softmax temperature $\zeta$ currently requires manual tuning; integration of intrinsic motivators (e.g., boredom, novelty) is pending.
- **Model expansion**: Online cloning/structural growth and related mechanisms are not dynamically addressed.
- **Physical embodiment**: Incorporation of FEPS and ECM update rules in real-world or neuromorphic agents remains future work.

FEPS thus establishes an interpretable, fully-internal-reward, graph-based instantiation of active inference, providing cognitive mapping, distinct hidden-state contextualization, and preference-adaptive policies through EFE minimization [2411.14991].

Source: https://www.emergentmind.com/topics/free-energy-projective-simulation-feps