Intrinsic Motivation Signals

• Intrinsic Motivation Signals are endogenous reward surrogates that drive exploration, skill acquisition, and model refinement in both biological and artificial agents.
• They leverage frameworks like empowerment, prediction error, and novelty bonuses to optimize behavior in high-dimensional and sparse environments.
• Practical implementations use RL methods (e.g., PPO, SAC) to balance exploration and exploitation, enhancing learning efficiency and task performance.

Intrinsic motivation signals are endogenous reward surrogates formalized to drive exploratory, skill-acquiring, or goal-flexible behavior in both biological and artificial agents in the absence of explicit extrinsic rewards. These signals function as dense, real-time surrogates for value alignment, option retention, and model improvement—supporting efficient exploration, complex task acquisition, and unsupervised skill learning across high-dimensional, stochastic, or open-ended behavioral domains.

1. Theoretical Foundations and Formal Definitions

Intrinsic motivation signals operationalize a range of information-theoretic and dynamical objectives, notably empowerment, prediction error, entropy maximization, information gain, and path diversity. Empowerment, as defined in the information-theoretic control literature, is the maximal mutual information $$C(x_0) = \max_{p(a_0^{T_e-1}|x_0)} I[X_{T_e}; A_0^{T_e-1}|x_0]$$—i.e., the channel capacity between an agent's action sequence and future world states from a given state $x_0$ (Tiomkin et al., 2022). This viewpoint is generalized in “constrained entropy maximization” formulations, where intrinsic drives are cast as maximum-entropy variational inference problems with constraint sets tailored to the agent’s structure and existential requirements (Kiefer, 5 Feb 2025). Alternative formalizations include path entropy (e.g., Maximum Occupancy Principle), mutual information between agent and world subspaces, and reduction of uncertainty or model error about hidden environment variables (Moreno-Bote et al., 15 Jan 2026, Andres et al., 2022, Haber et al., 2018).

2. Algorithmic Instantiations and Computation

Practical computation of intrinsic signals varies according to the underlying principle:

• Empowerment: For continuous-state dynamical systems, empowerment reduces to evaluating the Gaussian channel capacity $$\frac{1}{2}\log\det[I + \Sigma_\eta^{-1} F \Sigma_a F^\top]$$, where $F$ is the time-unfolded, locally-linearized sensitivity matrix; $\{\lambda_i\}$, its singular values, determine action-to-future-state informativeness. Greedy intrinsic policy selection entails computing local SVDs and applying water-filling (Tiomkin et al., 2022).
• Prediction-error–driven curiosity: Prediction error is measured between an agent’s world model and observed transitions. Agents are motivated to select actions (or goals) that maximally increase the expected future loss of their own forward or inverse models, e.g., $$r^I_t(a) = \sum_{s=1}^T \sum_{k=1}^{C_l} k p^{s}_{lm}(k|s_t,a)$$ (Haber et al., 2018, Martinez et al., 2023).
• Count-based and novelty bonuses: Intrinsic reward tracks visitation statistics—either via direct state/action counts ($r^i_t = \max (1/N(s_{t+1}) - 1/N(s_t), 0)$, or $r_i = 1/\sqrt{N(s)}$) or via density models in large/continuous spaces (Andres et al., 2022, Seurin et al., 2021).
• Successor feature control: Global state novelty is captured by the squared difference in successor features, $$r^{\mathrm{SFC}}_{t+1} = \|\psi_{\pi,\phi}(s_{t+1}) - \psi_{\pi,\phi}(s_t)\|_2^2$$; this detects bottlenecks and macro-novelty across trajectories (Zhang et al., 2019).
• Synergistic and composite signals: Joint action effects not reducible to single agents are rewarded by compositional prediction error, e.g., $$r_2^{\mathrm{int}}(s,a) = \|f^{\mathrm{joint}}(s,a) - f^{\mathrm{composed}}(s,a)\|_2$$ (Chitnis et al., 2020). Linear combinations, e.g., $x_0$0, balance breadth and directedness (Quadros et al., 25 Aug 2025, Martinez et al., 2023).

Optimization is typically performed through standard RL paradigms (PPO, SAC, DQN), maximizing the sum of extrinsic and intrinsic returns, with explicit scheduling or weighting to manage the exploration–exploitation balance (Andres et al., 2022, Zhang et al., 2019, Martinez et al., 2023).

3. Empirical Properties and Benchmark Results

Intrinsic motivation signals, when used as sole rewards, can induce sophisticated behaviors in benchmark control tasks absent any extrinsic signal. Notably, greedy empowerment control produces automatic swing-up and stabilization of canonical systems like the inverted pendulum and cart-pole, with trajectories and torque profiles resembling those derived from hand-crafted cost functions (Tiomkin et al., 2022). Controls based on controllable information production (gap between open-loop and closed-loop Kolmogorov–Sinai entropy) also achieve swing-up and “edge-of-chaos” regimes that require no extrinsic shaping (Shah et al., 30 Jan 2026). Count-based, curiosity-driven, or successor-feature signals enable deep exploration and effective sample efficiency in procedurally generated or sparse-reward environments (MiniGrid, VizDoom), often halving sample complexity or outperforming prediction-error–only baselines (Andres et al., 2022, Zhang et al., 2019, Seurin et al., 2021). LLM-guided signals further accelerate exploration by injecting goal-relevant, language-derived encouragement, improving learning curves and final success rates (Quadros et al., 25 Aug 2025).

In human-agent comparisons on open-world tasks (e.g., Crafter (Lidayan et al., 31 Mar 2025)), empowerment and entropy-based objectives most closely align with human exploration, with entropy driving early-stage state coverage and empowerment guiding advanced, controllable mastery. Information gain and immediate prediction error show weaker or degenerate correlation with human exploratory progress.

4. Unification and Generalization Across Intrinsic Drives

Information-theoretic frameworks subsume diverse intrinsic signals under a common principle: maximizing entropy or mutual information of action-state-paths, subject to constraints (e.g., homeostasis, controllability, bodily integrity). Empowerment, causal entropic forcing, and control Lyapunov sensitivity emerge as limit cases of channel-capacity maximization over variably-defined horizons and action-observation windows (Tiomkin et al., 2022, Kiefer, 5 Feb 2025). Controllable information production formalizes IM as the entropy-gap closed by agent feedback, generalizing “explore vs. exploit” as “seek and regulate chaos” (Shah et al., 30 Jan 2026).

Motivation-consistent intrinsic rewards can be constructed to align the policy gradients induced by intrinsic and extrinsic objectives, ensuring that internal signals directly support ultimate task progress regardless of sparse external feedback (Wang et al., 2022). Hierarchical RL and scheduling further dissociate “pure exploration” from “exploitation,” showing synergistic gains from alternating policies driven by distinct signals (Zhang et al., 2019).

5. Limitations, Domain Dependence, and Open Challenges

Intrinsic motivation signals are not universal panaceas. Prediction-error–based and count-based curiosity rapidly decay as models improve or state spaces saturate, risking disengagement; count-based novelty can drive “random walk” or degenerate behavior if extrinsic structure is absent or too delayed (Andres et al., 2022, Lidayan et al., 31 Mar 2025). Mechanistic signals (e.g., DoWhaM’s action usefulness) target rare but functionally critical opportunities but may overfit or stall if non-informative rare actions exist, or if high-frequency stochastic state changes “dilute” the effectiveness test (Seurin et al., 2021).

Empowerment and controllable-path objectives require accurate linearizations and noise-covariance estimates; in high dimensions, SVD and water-filling solutions become computational bottlenecks. World-model–based signals, including adversarial prediction error, hinge on the pretraining and stability of the prediction architecture (Martinez et al., 2023, Davoodabadi et al., 2024).

Human-aligned IM signals (as measured by prediction of human “interestingness” in physical domains) are best captured by forward-model–based adversarial metrics, possibly complemented with simple scene features (collisions, energy) (Martinez et al., 2023). However, all such models remain well below the behavioral reliability ceiling, indicating room for richer, multi-modal, or psycholinguistically grounded innovation, e.g., language-conditioned exploration or goal-verbalization scaffolding (Lidayan et al., 31 Mar 2025, Quadros et al., 25 Aug 2025).

6. Practical Design Considerations and Applications

Choice of intrinsic motivation signal should follow principled analysis of domain structure, reward sparsity, and embodiment constraints. Empowerment is recommended for domains where controllability and option-value are central, particularly for skill acquisition and stabilization (Tiomkin et al., 2022, Lidayan et al., 31 Mar 2025). Curiosity and prediction-error signals are most effective where model-improvement is central and when paired with suitable replay or self-imitation buffers (Andres et al., 2022, Haber et al., 2018). Action-usefulness and successor-feature signals are suitable for environments with compositional, agent-object interactions or bottlenecked transition graphs (Seurin et al., 2021, Zhang et al., 2019).

State-of-the-art implementations deploy these signals in a variety of RL architectures, from model-free on-policy PPO to off-policy replay buffers, coupled with explicit reward scaling, episodic regularization, and scheduled alternation between exploratory and exploitative drives.

The research trajectory points toward composite, dynamically scheduled, or learned intrinsic signals guided by human data, language, or demonstration (through IRL or meta-gradient alignment) and grounded in fundamental dynamical-system properties (Quadros et al., 25 Aug 2025, Martinez et al., 2023, Wang et al., 2022, Moreno-Bote et al., 15 Jan 2026).

---

References:

• "Intrinsic Motivation in Dynamical Control Systems" (Tiomkin et al., 2022)
• "Controllable Information Production" (Shah et al., 30 Jan 2026)
• "Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation" (Andres et al., 2022)
• "LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning" (Quadros et al., 25 Aug 2025)
• "Intrinsic motivation as constrained entropy maximization" (Kiefer, 5 Feb 2025)
• "Measuring and Modeling Physical Intrinsic Motivation" (Martinez et al., 2023)
• "Intrinsically-Motivated Humans and Agents in Open-World Exploration" (Lidayan et al., 31 Mar 2025)
• "Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness" (Seurin et al., 2021)
• "Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration" (Zhang et al., 2019)
• "Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards" (Wang et al., 2022)
• "Mutual Information State Intrinsic Control" (Zhao et al., 2021)
• "Intrinsic Motivation for Encouraging Synergistic Behavior" (Chitnis et al., 2020)
• "Emergence of Structured Behaviors from Curiosity-Based Intrinsic Motivation" (Haber et al., 2018)
• "Tracking Emotions: Intrinsic Motivation Grounded on Multi-Level Prediction Error Dynamics" (Schillaci et al., 2020)
• "How Intrinsic Motivation Underlies Embodied Open-Ended Behavior" (Moreno-Bote et al., 15 Jan 2026)