Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2304.02532v2)

Abstract: We propose a new policy representation based on score-based diffusion models (SDMs). We apply our new policy representation in the domain of Goal-Conditioned Imitation Learning (GCIL) to learn general-purpose goal-specified policies from large uncurated datasets without rewards. Our new goal-conditioned policy architecture "$\textbf{BE}$havior generation with $\textbf{S}$c$\textbf{O}$re-based Diffusion Policies" (BESO) leverages a generative, score-based diffusion model as its policy. BESO decouples the learning of the score model from the inference sampling process, and, hence allows for fast sampling strategies to generate goal-specified behavior in just 3 denoising steps, compared to 30+ steps of other diffusion based policies. Furthermore, BESO is highly expressive and can effectively capture multi-modality present in the solution space of the play data. Unlike previous methods such as Latent Plans or C-Bet, BESO does not rely on complex hierarchical policies or additional clustering for effective goal-conditioned behavior learning. Finally, we show how BESO can even be used to learn a goal-independent policy from play-data using classifier-free guidance. To the best of our knowledge this is the first work that a) represents a behavior policy based on such a decoupled SDM b) learns an SDM based policy in the domain of GCIL and c) provides a way to simultaneously learn a goal-dependent and a goal-independent policy from play-data. We evaluate BESO through detailed simulation and show that it consistently outperforms several state-of-the-art goal-conditioned imitation learning methods on challenging benchmarks. We additionally provide extensive ablation studies and experiments to demonstrate the effectiveness of our method for goal-conditioned behavior generation. Demonstrations and Code are available at https://intuitive-robots.github.io/beso-website/

• The paper introduces BESO, a novel approach using score-based diffusion models to achieve fast, multi-modal goal-conditioned imitation learning in just three denoising steps.
• BESO simplifies policy architectures by unifying goal-dependent and goal-independent tasks without relying on hierarchical designs.
• Extensive benchmarks on Relay Kitchen and Block-Push tasks validate BESO's superior performance and efficiency over existing methods.

Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

The paper by Moritz Reuss et al., entitled "Goal-Conditioned Imitation Learning using Score-based Diffusion Policies," introduces a novel approach to Goal-Conditioned Imitation Learning (GCIL) by leveraging score-based diffusion models (SDMs). The authors propose a goal-conditioned policy architecture termed "BEhavior generation with ScOre-based Diffusion Policies" (BESO), which is structurally distinct from existing models like Latent Plans or C-BeT.

Key Contributions

1. Score-Based Diffusion Policies in GCIL: BESO leverages the expressive power of SDMs to capture multi-modal distributions within play data. The approach decouples the score model learning from the inference sampling process, enabling fast action generation in merely three denoising steps. This contrasts significantly with other diffusion-based policies that typically require over thirty steps.
2. Simplified Architecture: Unlike hierarchical models, BESO does not rely on complex or compartmentalized systems to achieve effective goal-conditioned behavior learning. This simplicity in design does not come at the cost of performance, as BESO can extend its applicability to both goal-dependent and goal-independent tasks using classifier-free guidance within a single framework.
3. Benchmarking and Performance: Through extensive benchmarking, the paper illustrates that the BESO framework consistently surpasses state-of-the-art GCIL methods on demanding tasks, such as those found in conditioned Relay Kitchen and Block-Push environments. BESO's applicability is further validated by ablation studies that emphasize its efficiency and broader adaptability to different simulation environments.

Methodology

BESO applies a novel score-based generative diffusion model to learn goal-conditioned policies from unstructured play data. The framework adopts a continuous diffusion process mapped by an SDE, progressively transforming samples from play data into noise. The expressiveness inherent to this method enables a comprehensive capture of behavior policy distributions—fundamental for GCIL.

Implications and Future Directions

BESO represents a significant stride in leveraging SDMs for robotic learning tasks that require imitation from complex datasets. By reducing the dependence on highly structured environment interactions or reward-based systems, BESO suggests potential for more autonomous robot training paradigms. The paper also opens avenues for further research into classifier-free guidance for policy learning and the use of SDMs for more generalized AI solutions.

In conclusion, the paper provides a substantial technical contribution to the field of robotics and artificial intelligence by showcasing how diffusion-based techniques can enhance policy learning from multimodal interaction data. Future research may further explore parameter tuning of diffusion processes, the scalability to real-world environments, and integrations with other AI-learning frameworks.