Knowledge Reutilization in Meta-Reinforcement Learning

Abstract: Meta-reinforcement learning enables fast adaptation by extracting shared structure from related tasks, but existing end-to-end methods often couple task inference with embodiment-specific control. This coupling can obscure non-parametric task semantics, reduce sample efficiency, and limit cross-agent reuse. We propose a meta-knowledge reutilization framework that learns task-level knowledge on a dynamics-simplified agent and transfers it to heterogeneous agents. The framework uses a Bayesian non-parametric prior to organize latent task modes and a high-level policy to generate task-level magnitude guidance. To bridge reusable task knowledge with different embodiments, we introduce a semantic-magnitude interface and a lightweight temporal adaptor, which convert frozen meta-knowledge into temporally aligned subgoals for embodiment-specific low-level controllers. Experiments on multiple locomotion agents show that our framework reduces final-step tracking error by 94.75% -- 99.79% compared with recent state-of-the-art baselines and achieves comparable deployment performance with about 23.8% of their interaction data.

• The paper introduces ReMAP, a framework that decouples abstract task understanding from low-level policy learning using a DPMM prior and dual learning branches.
• It demonstrates that ReMAP outperforms baseline methods with up to 99.79% reduced tracking error and superior sample efficiency across diverse agents.
• The framework employs a semantic-magnitude interface and stride predictor to align temporal dynamics, ensuring robust policy deployment in heterogeneous simulated environments.

Authoritative Summary of "Knowledge Reutilization in Meta-Reinforcement Learning" (2606.18132)

Motivation and Framework Overview

The paper introduces ReMAP, a meta-reinforcement learning (Meta-RL) framework designed to decouple task-level meta-knowledge acquisition from embodiment-specific policy learning. The motivation stems from the inefficiencies and transfer limitations of existing end-to-end Meta-RL approaches, where task inference is entangled with agent-specific dynamics and often relies on restrictive latent priors (typically a fixed Gaussian). This coupling obscures task semantics and hampers knowledge reuse across heterogeneous agents. ReMAP addresses: (1) learning non-parametric task representations via a Dirichlet Process Mixture Model (DPMM) prior on a dynamics-simplified agent, (2) disentangling task inference from low-level control using dual independent learning branches, and (3) deploying meta-knowledge across diverse agents using a semantic-magnitude interface (SMAI) and a stride predictor for temporal alignment.

Figure 1: The ReMAP framework learns meta-knowledge on a simplified agent, adapts low-level policies via magnitude-guided curriculum, and reuses frozen meta-knowledge for deployment on heterogeneous agents.

Methodological Contributions

ReMAP's design centers on disentanglement and modularity:

• Task-Level Meta-Knowledge Acquisition: The inference module (implemented as a VAE with DPMM prior) learns abstract task representations from trajectory contexts consisting of position, velocity, and reward, using a context window for history compression. The DPMM (stick-breaking construction) flexibly organizes latent task modes, enabling adaptive clustering without predefining component counts. The high-level policy operates in the abstract task space, translating latent semantics into magnitude guidance. Both modules are trained exclusively on a simplified agent with minimal dynamics—ensuring that the latent encodings capture task structure rather than agent-dependent control patterns.

Figure 2: ReMAP workflow—disentangled training on the simplified agent with DPMM-based inference and independent low-level policy warmup via magnitude curriculum; deployment transfers frozen meta-knowledge through SMAI and stride predictor.

• Magnitude-Triggered Disentangled Low-Level Policy Learning: Each complex agent trains its low-level controller independently. SMAI bridges the semantic guidance channel (e.g., position or velocity) with a scalar magnitude signal. A curriculum schedule gradually increases the complexity (magnitude, task switch frequency, horizon, initial diversity) for curriculum-driven adaptation. The objective is strictly to learn reliable subgoal tracking under the agent's own dynamics.

  Figure 3: Convergence of reward for meta-knowledge acquisition; high-level policy and inference module accurately track position and velocity targets; t-SNE reveals well-separated latent task clusters.

• Deployment and Temporal Adaptation: When deployed, the target agent projects its state into the abstract task space, infers latent task mode via the frozen module, and receives magnitude guidance. The stride predictor estimates execution duration for each subgoal, aligning temporal semantics across agents with differing inertial and morphological properties.

Empirical Evaluation

The experiments comprehensively assess cross-embodiment knowledge transfer in the Mujoco environment with Hopper, Walker, Half-Cheetah, and Ant. The task suite includes both goal-reaching and velocity-tracking in forward and backward directions.

Figure 4: Evaluation agents—distinct morphologies (Hopper, Walker, Half-Cheetah, Ant)—each tested on four locomotion tasks.

• Meta-Knowledge Acquisition: Training curves exhibit steady convergence of reward for the simplified agent. The latent variables, visualized via t-SNE, form distinct clusters, validating DPMM-driven semantic separation. Inference trajectories show accurate tracking of target positions and velocities.
• Disentangled Low-Level Policy Learning: Each agent achieves stable locomotion and magnitude tracking under SMAI conditioning, though agents with less stable morphologies require longer training.
• Cross-Agent Meta-Knowledge Reutilization: Frozen inference modules and high-level policies generalize to all target agents. Across the 12 non-parametric tasks, agents achieve consistent goal and velocity tracking, irrespective of embodiment. t-SNE visualizations post-deployment show semantic clusters in the latent space are preserved. Confusion matrices reveal decoder prediction accuracy of latent task modes remains high, substantiating embodiment invariance.

Figure 5: Cross-agent deployment—trajectory tracking, t-SNE latent cluster preservation, decoder confusion matrices—validating semantic consistency and prediction accuracy across embodiments.

Comparative and Quantitative Results

• Baseline Comparison: ReMAP is benchmarked against RL$^2$ (recurrence-based), PEARL (probabilistic context-based), CEMRL (GMM-based), and MELTS (DPMM-based, end-to-end). ReMAP consistently achieves lower tracking MSE (goal and velocity), especially at final rollout timesteps, with reductions of 94.75%–99.79% relative to baselines.
• Sample Efficiency and Training Cost: ReMAP attains comparable deployment performance with approximately 23.8% of MELTS's interaction data (38M vs. 160M steps). Ablative studies demonstrate that fixed Gaussian priors and improper GMMs degrade semantic separation, underscoring the necessity of Bayesian non-parametric priors.

Implications and Future Directions

ReMAP establishes that disentangled task-level meta-knowledge, acquired with expressive and adaptive priors, is reusable across varied embodiments without retraining. This decoupling advances sample-efficient cross-embodiment adaptation, reducing transfer complexity and promoting modular RL architectures. The implications are substantial for robotics: agents can rapidly adapt to novel morphologies by leveraging transferable abstract task knowledge. Theoretical advances in Bayesian non-parametric clustering synergize with practical modularization for scalable policy deployment.

Future research will explore expanded semantic-magnitude interfaces, broader task diversity, and sim-to-real transfer to bridge the gap between simulation-based abstraction and physical deployment.

Conclusion

ReMAP achieves efficient cross-embodiment policy transfer in Meta-RL via task-level meta-knowledge regularized by DPMM and disentangled from low-level control. The framework delivers robust tracking accuracy, high semantic separability, and abated training cost, outperforming end-to-end baselines and demonstrating the practical and theoretical utility of reusable abstract task representations.