Curiosity-Driven Exploration

• Curiosity-Driven Exploration is a paradigm where intrinsic rewards from novelty and learning progress drive agents to explore sparse-reward environments using disentangled representations.
• It employs modular techniques including learned goal spaces and projection operators to isolate controllable features and boost exploration efficiency.
• CDE advances robotics and reinforcement learning by enabling adaptive skill discovery, effective transfer learning, and structured exploration in high-dimensional settings.

Curiosity-Driven Exploration (CDE) is a paradigm in machine learning, control, and reinforcement learning that leverages intrinsic motivation—formalized as measures of novelty, surprise, or learning progress—to drive agents to independently and efficiently discover diverse behaviors and solutions in complex, often sparse-reward environments. CDE frameworks employ a wide variety of algorithmic constructs including learned goal spaces, representation disentanglement, modular exploration strategies, information-theoretic progress metrics, and integration with adaptive curricula or model-based planning to achieve scalable, robust, and generalizable exploration capabilities.

1. Representation of Goal Spaces: Disentangled vs. Entangled Structures

A central tenet in curiosity-driven goal-directed exploration is the structure of the learned goal space. Disentangled goal spaces, typically instantiated via $\beta$-Variational Autoencoders ($\beta$-VAEs), produce latent representations $z = [z_1, \dots, z_n]$ in which each latent dimension $z_i$ corresponds to a distinct, independent generative factor—such as the position of a specific object in the environment. This modularity in the goal space mapping allows for projection operators $P_k R(o)$ that isolate particular controllable factors. Entangled representations, by contrast, blur this correspondence and embed multiple environmental factors into each latent variable, which inhibits targeted control and modular goal sampling.

The empirical advantage of disentangled representations is evidenced in robotic manipulation tasks (e.g., the Arm-2-Balls environment), where modular goal exploration driven by a disentangled latent space leads to higher exploration coverage and more efficient discovery of independently controllable objects, yielding performance that matches or exceeds approaches based on engineered features. In modular Intrinsically Motivated Goal Exploration Processes (IMGEPs), the goal sampling policy factorizes as $\gamma(\tau) = \gamma(\tau \mid i) \cdot p(i)$, where $p(i)$ is an adaptively updated interest measure concentrating sampling in modules that demonstrate rapid learning progress.

2. Learning Progress and Measures of Curiosity

Curiosity-driven exploration formalizes the intrinsic reward signal as a function of learning progress or competence progress within the agent’s learned state or goal space. Given a representation module $R$ and modular decomposition indexed by $i$, the learning progress at time $t$ is: $$\delta_t^{(i)} = C_i(\tau_t, P_i R(o^{\prime})) - C_i(\tau_t, P_i R(o_t))$$ where $C_i$ is a cost function (e.g., Euclidean distance) between the attempted goal and the projected latent representation of observed states. The moving average of $\delta_t^{(i)}$ forms the “interest measure” $\Upsilon_i(t)$, which updates the sampling probability via: $$p(i) = 0.9 \cdot \frac{\Upsilon_i(t)}{\sum_k \Upsilon_k(t)} + 0.1 \cdot \frac{1}{N}$$ This heuristic places exploration priority on modules with the greatest learning velocity, focusing agent attention on learnable, controllable features while filtering out distractors or uncontrollable facets. The meta-policy thereby implicitly discovers independently controllable latents in an unsupervised manner.

3. Algorithmic and Architectural Considerations

The practical realization of CDE frameworks involves several architectural choices:

• Representation learning: Training a deep encoder $R: \mathcal{O} \rightarrow \mathbb{R}^n$ (e.g., via $\beta$-VAE) to obtain a latent goal space with high disentanglement.
• Modular decomposition: Constructing projection operators $P_k$ to isolate latent dimensions for modular goal sampling.
• Curiosity computation: Online monitoring of learning progress in each module, which not only drives goal selection but signals which environmental features are independently controllable.
• Exploration policy: Incorporating curiosity-driven reward/interest signals into the goal sampling process, then inverting the mapping to generate low-level control commands to achieve selected goals.

Empirical findings demonstrate:

Approach	Sample Efficiency	Diversity of Exploration	Sensitivity to Distractors
Modular, Disentangled	High	High	Low
Non-modular, Entangled	Lower	Lower	High

When exploration noise is low, only modular curiosity in a disentangled space reliably produces sustained, structured exploration along controllable degrees of freedom.

4. Robotics and Applied Implications

CDE methodologies trained with disentangled representations extend immediately to robotics and complex multi-object environments. In physical robotic settings (e.g., manipulation of several objects with some under the agent’s control and others acting as distractors), modular CDE allows the agent to focus skill acquisition and exploration exclusively on apparatus that it can manipulate, ignoring extraneous dynamics. Performance in state or goal space coverage using modular, disentangled CDE approaches is empirically shown to match the efficiency of strategies relying on manually engineered goal features, but with greater adaptability to unknown or changing environmental structure.

5. Discovery of Controllable Features and Abstraction

A notable emergent property of the learning progress measure—used as the core curiosity metric—is its alignment with the discovery of independently controllable features. As the agent’s representation of the environment becomes increasingly disentangled and modular, the modules with sustained or increasing learning progress identify the actionable subspaces of the environment. This discovery is achieved without direct supervision, forming a basis for higher-level abstractions and skill hierarchies. The knowledge of feature controllability, once surfaced during CDE, potentially facilitates:

• Construction of reusable skill libraries
• Transfer learning to new environments (particularly in sim-to-real transfer)
• Building compact, interpretable policy abstractions for complex, multi-object tasks

6. Limitations and Prospects for Future Work

The principal limitation cited is the representation learning phase, which is typically performed offline, thereby limiting adaptation as new environmental features emerge. Moving toward online representation learning—where disentanglement and modularity are adjusted in synchrony with active exploration—represents a promising direction for overcoming this constraint.

Additionally, integrating the mechanisms for discovering controllable latents with subsequent hierarchical planning, transfer learning, or multi-task skill acquisition could significantly improve the efficiency and versatility of autonomous agents.

Continuation in the incorporation of learning progress signals both as curiosity drives and as proxies for controllability offers a principled path toward scalable, autonomous skill discovery and efficient exploration in high-dimensional, sparse-reward settings.

7. Summary

Curiosity-Driven Exploration in learned, disentangled goal spaces leverages the modularity and independence of latent space factors—acquired via deep representation learning algorithms—to guide efficient and comprehensive exploration. By coupling modular goal sampling with interest-driven (learning-progress) intrinsic rewards, agents autonomously prioritize learning on segments of the environment that afford independent control, even in the presence of distractors or high-dimensional, continuous actions. The architecture reliably surfaces controllable features within the latent space, yielding efficient exploration performance in complex environments, and points toward robust directions for lifelong learning, transfer, and hierarchical skill development.