Entropy-Controlled Intrinsic Motivation

• ECIM is a framework that modulates an agent's internal entropy to balance exploration and exploitation, enabling more adaptive behavior.
• It employs adaptive techniques like UCB-based surprise selection and dynamic entropy scheduling to switch between curiosity-driven and stabilization policies.
• Empirical results show that ECIM enhances state coverage, skill diversity, and sample efficiency in both simulated and robotic environments.

Entropy-Controlled Intrinsic Motivation (ECIM) refers to a class of intrinsic motivation frameworks for artificial agents, in which the agent’s policy is guided explicitly by controlling, maximizing, or minimizing entropy-related signals derived from its own internal models or reward structure. ECIM agents adapt their exploration and exploitation dynamically depending on entropy estimates of their sensory, belief, or state visitation distributions. This principle unifies distinct algorithmic paradigms—curiosity-driven exploration (entropy maximization), niche-seeking or stabilization (entropy minimization), and information-theoretic optimal control—by treating entropy not as a fixed objective but as a flexible control variable.

1. Theoretical Foundations and Objective Formulations

At the core of ECIM methods is the optimized management of entropy in an agent’s behavioral or representational distributions. Let $d^{\pi}(s)$ denote the state-marginal under policy $\pi$, and $p_{\theta_t}(s)$ be the agent’s learned estimate of that marginal at time $t$. The Shannon entropy is given by

$$\mathcal{H}\bigl(p\bigr) = - \mathbb{E}_{s\sim p} [\log p(s)].$$

Two canonical intrinsic rewards arise:

• Entropy-minimization: $$r_{\mathrm{int}}^{\min}(s_t, a_t) = \log p_{\theta_t}(s_{t+1})$$ (drives policy toward predictable, controllable, low-entropy states)
• Entropy-maximization: $$r_{\mathrm{int}}^{\max}(s_t, a_t) = -\log p_{\theta_t}(s_{t+1})$$ (encourages novelty, curiosity, and high-entropy exploration) Both are unified as one-step surprise objectives with a sign flip.

A general ECIM agent can formulate its objective as a constrained max-entropy problem: $$\mathcal{L}(Q) = -H[Q(\pi)] + \lambda_1\, \mathbb{E}_{Q(\pi),Q(s|\pi)} [-\ln P(o,s)] + \lambda_2\, D_{KL}(Q(s|o,\pi) || Q(s|\pi)),$$ where $\lambda_1$ and $\lambda_2$ balance exploitation (homeostasis, model-evidence constraint) and exploration (information gain, empowerment) (Kiefer, 5 Feb 2025).

2. Adaptive Entropy Control Mechanisms

Purely maximizing or minimizing entropy is suboptimal in diverse environments, since high-entropy strategies can fail in settings requiring stabilization, while entropy minimization stalls in inherently stochastic regimes. ECIM frameworks incorporate adaptive mechanisms to mediate between these extremes:

Surprise-Adaptive Intrinsic Motivation (SAIM):

The reward selection process is cast as a 2-armed bandit problem: the agent decides each episode between $r^{\min}$ and $r^{\max}$ using an Upper Confidence Bound (UCB) selector,

$$\alpha^{(m)} = \arg\max_{i \in \{0,1\}} \mu_i^{(m)} + \sqrt{c \log m / N_i^{(m)}},$$

where $\mu_i^{(m)}$ tracks mean feedback for arm $i$ (Hugessen et al., 27 May 2024). The feedback signal measures controllable entropy change: $$f_m = \left| \frac{\mathcal{H}(p_{\theta_T}^{(m)}) - \mathcal{H}(p_{\theta_T}^{\mathrm{rand}})}{\mathcal{H}(p_{\theta_T}^{\mathrm{rand}})} \right|.$$ The agent updates its preference toward the objective delivering the highest deviation from random entropy, flexibly favoring stabilization or curiosity as appropriate.

Adaptive Entropy Scheduling (AECPOM):

Alternatively, the entropy bonus coefficient $\beta_t$ in PPO-style policy optimization is tied to agent performance: $\beta_t = \beta_{\max} \frac{R_t}{R_{\max}},$ with $R_t$ the moving-average return. Early training maintains high $\beta_t$ (exploration); as $R_t$ increases, $\beta_t$ decays, allowing a shift toward exploitative, lower-entropy policies (Gong et al., 6 Dec 2025).

3. Constrained Intrinsic Motivation and Skill Discovery

Recent advances cast ECIM objectives as constrained entropy maximization for sample-efficient skill discovery and unbiased exploration:

• In reward-free pre-training (RFPT), the policy $\pi(a|s,z)$ and state encoder $\phi$ are optimized to maximize the conditional entropy $H(\phi(s')|z)$ while enforcing skill alignment through contrastive NCE bounds (Zheng et al., 12 Jul 2024). The scalar reward is estimated via a $k$-NN density estimator in the projected latent subspace,

$$r^{\mathrm{int}}(s, z) = \log\left(1 + \frac{1}{\xi} \sum_{j=1}^{\xi} |g_z(\phi(s)) - g_z(\phi(s))^j|\right),$$

where $g_z(\phi(s)) = \phi(s)^\top z$.

• When extrinsic rewards are made available, an adaptive dual gradient controls the intrinsic-extrinsic balance via the Lagrangian multiplier $\lambda$ and adaptive coefficient $\tau_k^{\mathrm{CIM}}$ to prevent intrinsic rewards from biasing the final policy (Zheng et al., 12 Jul 2024).

Empirical results demonstrate that ECIM agents outperform fixed-objective and baseline methods in state coverage, skill diversity, and fine-tuning, particularly in high-dimensional robotics domains.

4. Latent State-Space and Information-Theoretic Realizations

ECIM can be instantiated via latent state-space models that define entropy not directly on observations but on learned low-dimensional representations:

• In partially observed Markov processes, agents learn variational posteriors $q_t(z)$ over latents $z_t$ via deep variational Bayes filters (Rhinehart et al., 2021). The control objective is to minimize the latent-state visitation entropy,

$$J_{\mathrm{ECIM}}(\pi) = -H\bigl(d^\pi(z)\bigr) = -\sum_{z} d^{\pi}(z) \log d^{\pi}(z),$$

where $d^\pi(z)$ is approximated as the time-averaged mixture of $q_t(z)$. The ECIM intrinsic reward is then the negative KL divergence $-\mathrm{KL}(q_t(z) || q(z))$.

• Empirical evaluations indicate that such intrinsic control reliably induces agents to both observe dynamic features and stabilize them, e.g., by capturing moving objects or reducing environmental unpredictability (niche creation).

5. Algorithmic Pseudocode and Implementation

Common ECIM algorithmic templates follow a loop integrating entropy management, policy learning, and intrinsic reward computation:

initialize policy π, entropy estimator/model, tuning coefficients for each episode: select intrinsic objective (entropy minimization, maximization, or adaptive) collect trajectory under current policy update policy π via RL algorithm (e.g., DQN, PPO) using total (extrinsic + intrinsic) reward compute entropy/feedback signal (change in state-marginal or latent entropy) update selection statistics or entropy control coefficients (e.g., UCB, β_t, λ, τ) periodically evaluate controllable entropy against random baselines end for

Specific instantiations adjust the selection logic: UCB bandit selection (Hugessen et al., 27 May 2024), dual-gradient updates for constraints (Zheng et al., 12 Jul 2024, Kiefer, 5 Feb 2025), or AECPOM for entropy coefficients (Gong et al., 6 Dec 2025).

6. Empirical Results and Practical Impact

Experimental studies across gridworlds, Arcade Learning Environment benchmarks, MuJoCo robotics, and quadrupedal robot locomotion converge on several consistent findings:

Environment	ECIM Variant	Key Result	Entropy Change vs. Random
Maze (low entropy)	S-Max/S-Adapt	0.85 avg. return	+120%
Tetris (high entropy)	S-Min/S-Adapt	180 steps survival	–45%
Freeway (Atari)	S-Adapt	24 (75% extrinsic)	+75%

In complex robot tasks, ECIM produces:

• 4–12% higher cumulative rewards, 23–29% lower pitch oscillation, 20–32% less joint acceleration, and 11–20% lower torque consumption in ANYmal quadruped across six terrains (Gong et al., 6 Dec 2025).
• Substantially improved state-space coverage, dynamic skill diversity, and sample efficiency (up to 20× gain) in MuJoCo and URLB Walker suite, outperforming 15 baseline methods (Zheng et al., 12 Jul 2024).

These gains are attributed to ECIM's principled ability to modulate exploration/exploitation pressure, prevent premature convergence, and structure emergent behavior via entropy-aware feedback mechanisms.

7. Broader Implications, Limitations, and Extensions

ECIM acts as a conceptual umbrella unifying classic and modern intrinsic motivation paradigms (curiosity, empowerment, active inference, maximum-occupancy principle) under the lens of constrained entropy control (Kiefer, 5 Feb 2025). By tuning the Lagrangian multipliers or adaptive schedules, ECIM systems interpolate smoothly between information-seeking, stabilization, and homeostasis.

Limitations include dependence on the representational capacity of the underlying world models (especially in high-dimensional or highly stochastic regimes), potential for degenerate solutions (e.g., collapsing entropy by extinguishing environmental diversity), and the challenge of extending to multi-agent and safety-critical domains. Theoretical connections to the free-energy principle and rigorous understanding in partially observed or non-stationary environments remain open.

Ongoing research is advancing ECIM in real robotics, reward-free pre-training, and multi-agent systems, with active developments on safety mechanisms, more expressive latent state-spaces, and context-sensitive entropy modulation (Hugessen et al., 27 May 2024, Zheng et al., 12 Jul 2024, Gong et al., 6 Dec 2025, Rhinehart et al., 2021).