Self-Improvement via On-Manifold Exploration

• The paper introduces SOE, a method that confines exploration to a low-dimensional manifold to improve sample efficiency and safety.
• It integrates compact latent encoding with variational information bottleneck techniques to ensure valid, controlled action proposals.
• Empirical results demonstrate enhanced performance in robotic tasks, with notable improvements in exploration smoothness and reduced unsafe behaviors.

Self-Improvement via On-Manifold Exploration (SOE) refers to a class of methodologies that enable intelligent agents—ranging from statistical learners to robotic systems—to enhance their performance by restricting exploration to a structured, task-relevant manifold, rather than the ambient high-dimensional or unconstrained space. SOE is motivated by the need for efficient, safe, and effective exploration in the context of self-improvement, where the agent iteratively refines its policy, latent representation, or solution set by focusing exploration on actions or behaviors deemed valid or promising within a learned manifold. The approach encompasses compact latent representations, manifold learning, and constrained optimization, and is supported by both theoretical and empirical results showing superior sample efficiency, performance, and safety in applications such as robotic manipulation and LLM reasoning.

1. Theoretical Foundations and Motivation

SOE arises from the challenge that random or unconstrained perturbative exploration often leads to mode collapse, unsafe behaviors, and sample inefficiency—particularly in robotic control and high-dimensional optimization settings (Jin et al., 23 Sep 2025). Classic techniques for exploration (e.g., random noise injection) do not account for the structure of valid actions or policies, frequently inducing erratic trajectories and suboptimal data collection. Conversely, restricting exploratory actions to a data-driven or task-informed manifold exploits the intrinsic geometry and semantics of the domain, allowing the agent to generate diverse yet valid behaviors.

The central theoretical insight is that the set of skillful or semantically meaningful actions often forms a low-dimensional manifold embedded in the ambient control or solution space. By discovering, parameterizing, and confining exploration to this manifold, SOE aligns exploration with the agent’s competence boundaries, preserves safety constraints, and amplifies the informativeness of each trial.

2. Learning Compact Manifold-Constrained Latent Representations

A core mechanism in SOE is the learning of compact latent representations that capture only the task-relevant degrees of freedom, thus defining the exploration manifold. This is typically achieved using the variational information bottleneck (VIB) principle, which optimizes for high mutual information between the latent variable $Z$ and the action $A$ (ensuring $Z$ is predictive of valid actions), while minimizing mutual information between $Z$ and the raw observation $O$ (excluding irrelevant factors):

$\max_{\theta} I(Z; A) - \beta I(Z; O)$

The practical loss for this objective, when parameterized with neural networks for the encoder $p_{\theta}(Z|O)$ and the decoder $q_{\phi}(A|Z)$, is

$$\mathcal{L}_{\mathrm{IB}}(\theta, \phi) = \mathbb{E}_{(o,a)\sim\mathcal{D}} \Big[ -\mathbb{E}_{z\sim p_\theta(z|o)} \log q_\phi(a|z) + \beta\, \mathrm{KL}(p_\theta(Z|o) \,\|\, r(Z)) \Big]$$

where $\mathcal{D}$ denotes the demonstration or experience dataset and $r(Z)$ is a prior, often Gaussian. The resulting latent space forms the domain for on-manifold exploration, ensuring that perturbations in $Z$ yield behaviors consistent with the agent’s competence and operational validity (Jin et al., 23 Sep 2025).

3. Plug-in Architecture and Integration with Policy Models

SOE is implemented as a plug-in module that operates alongside arbitrary policy architectures, including diffusion policies and imitation learning pipelines. The architecture is dual-path:

• Base Path: Processes observations through conventional encoders for stable and accurate imitation-driven policy execution.
• Exploration Path: Projects observations into the compact latent space, samples perturbed $z$ with tunable variance (controlled by a hyperparameter $\alpha$), and decodes $z$ to action proposals using the action decoder. The VIB-induced constraints ensure that sampled explorations remain plausible and safe.

Only the exploration path is optimized with the VIB loss, maintaining the integrity and reliability of the base policy. This modularity enables seamless augmentation of existing systems without degrading baseline performance or introducing uncontrolled behaviors. The exploration path proposes rollouts that can be evalued, selectively incorporated, or further fine-tuned, providing the actionable data for self-improvement (Jin et al., 23 Sep 2025).

4. Human-Guided and Automated Steering of Exploration

The structure of the learned latent space enables both automated and human-guided exploration. Task-relevant latent dimensions can be identified using the signal-to-noise ratio (SNR):

$$\mathrm{SNR}_i = \mathrm{Var}(\mu_i) / \mathbb{E}[\sigma_i^2], \quad i = 1, \ldots, d$$

where $(\mu_i, \sigma_i^2)$ are the mean and variance of the $i$-th dimension of the latent variable $Z$ under the variational posterior $p_\theta(Z|O)$. High SNR dimensions capture salient, disentangled factors such as spatial axes in manipulation; users can selectively induce exploration along these axes, employing farthest point sampling to maximize behavioral diversity within safe, meaningful boundaries.

This enables application-specific, targeted exploration and supports interactive modalities where human operators can direct the agent’s curiosity and experiment with variations, further amplifying sample efficiency and transferability (Jin et al., 23 Sep 2025).

5. Experimental Validation in Simulation and Real-World Tasks

Extensive empirical results establish the effectiveness of SOE in both simulated benchmarks (e.g., Mug Hang, Toaster Load, Lamp Cap) and real robotic systems. Quantitative findings include:

• Superior success rates and Pass@k (multi-attempt success metrics) relative to baseline diffusion policies and prior methods such as SIME.
• Dramatic reductions in the number of required rollouts for successful trajectory collection, indicating improved sample efficiency and faster iteration-to-skill acquisition.
• Improvement in exploration smoothness and safety, measured via average jerk (lower values indicating less erratic, more physically plausible motions).
• Enhanced controllability and efficiency under human-guided exploration, with observed success rate improvements ranging from +19.1% to +62.0% in real-world settings.

The plug-in SOE module consistently avoids the unsafe, unstable behaviors characteristic of random full-space perturbations, thereby facilitating robust, reliable self-improvement (Jin et al., 23 Sep 2025).

6. Advantages over Random or Unconstrained Exploration

SOE’s restriction of exploration to a learned task-relevant manifold addresses several frequently observed pitfalls in unconstrained schemes:

• Safety: Actions remain within the valid operational domain, preserving physical constraints and system integrity.
• Exploration Efficiency: Each rollout is more likely to yield informative, actionable feedback, reducing the need for redundant or harmful trials.
• Diversity without Mode Collapse: By maintaining latent space diversity and enabling guided selection, SOE mitigates mode collapse (the convergence to a narrow set of behaviors).
• Compatibility and Modularity: The plug-in design and dual-path training permit retrofitting to a broad array of policy architectures, including diffusion and transformer-based agents.

This methodological advance establishes on-manifold exploration as a principled, general-purpose approach to self-improvement in policy-based agents and robotic control systems (Jin et al., 23 Sep 2025).

7. Mathematical Summary and Implementation

The SOE methodology can be concisely described by the following components:

Component	Mathematical Formulation	Purpose
VIB Latent Learning	$\max_{\theta} I(Z; A) - \beta I(Z; O)$	Compact, task-relevant latent encoding
Loss Function	$\mathcal{L}_{\mathrm{IB}}(\theta, \phi)$ (as above)	Tractable, variational optimization
Exploration Action Sampling	$\mu_t, \sigma_t \sim p_\theta(Z|O = o_t)$	Parametric posterior in latent space
	$z_t \sim \mathcal{N}(\mu_t, (\alpha \sigma_t)^2)$, $a_t \sim q_\phi(A|Z)$	Controlled, stochastic perturbation
Joint Training Objective	$$\mathcal{L}(\theta, \phi, \psi) = \mathcal{L}_{\mathrm{IL}}(\psi) + \mathcal{L}_{\mathrm{IB}}(\theta, \phi)$$	Decoupled update for exploration/base
Latent Steering Metric	$$\mathrm{SNR}_i = \mathrm{Var}(\mu_i) / \mathbb{E}[\sigma_i^2]$$	Feature disentanglement, human guidance

This framework enables highly efficient, safe, and adaptive robot self-improvement, with explicit mechanisms for integrating domain knowledge and operator preferences.

---

SOE thus unifies principled latent representation learning, modular policy augmentation, and task-aligned exploration to substantially improve both the efficacy and safety of self-improving agents in complex, real-world environments, with demonstrated advantages in sample efficiency and operational robustness (Jin et al., 23 Sep 2025).