Generative Image as Action Models (2407.07875v2)

Abstract: Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state-of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

• The paper introduces GENIMA, a framework that reformulates joint-action generation as an image-generation task using diffusion models.
• It employs a two-stage process with a fine-tuned Stable Diffusion model and an ACT controller to convert visual targets into joint positions.
• The evaluation across simulated and real-world tasks shows robust performance, outperforming state-of-the-art methods under scene perturbations.

A Formal Overview of "Generative Image as Action Models"

The paper "Generative Image as Action Models" presents an innovative exploration of using image-generation diffusion models for visuomotor control, introducing a framework named GENIMA. This work builds on the capability of diffusion models to generate high-fidelity images and extends their application beyond traditional domains such as image-editing and novel view synthesis. It leverages Stable Diffusion, fine-tuning it to "draw joint-actions" by interpreting actions as visual patterns on RGB images to control a sequence of joint positions.

Main Contributions

1. Problem Formulation: The authors present a novel problem formulation that reframes joint-action generation as an image-generation task. By casting actions in the image space, GENIMA utilizes the prowess of pre-trained diffusion models to derive action-patterns, eschewing the traditional reliance on 3D depth, keypoints, or motion planning.
2. Empirical Evaluation: In a comprehensive evaluation across 25 simulated RLBench tasks and 9 real-world manipulation tasks, GENIMA demonstrated considerable robustness, outperforming state-of-the-art visuomotor approaches like ACT and DiffusionPolicies in several tasks. Notably, GENIMA achieved superior robustness to scene perturbations and generalization to novel objects despite the absence of depth information.
3. Techniques Employed: The approach employs a two-stage process: firstly, leveraging ControlNet to fine-tune Stable Diffusion for drawing target joint positions on input images, and secondly, using an ACT controller to translate these targets into executable joint-positions. This ensures that the semantic and task-level reasoning is offloaded to the diffusion model, while the controller handles spatial execution.

Key Findings and Results

1. GENIMA achieved a 49.6% success rate on the RLBench tasks, indicating its competitive performance relative to state-of-the-art diffusion policies and robust response to contextual variations.
2. The ability of GENIMA to perform comparably to 3D-focused agents in tasks involving non-linear trajectories and tiny objects reflects its effectiveness even without 3D priors.
3. The robust performance across diverse perturbations (e.g., object color, lighting changes) substantiates the potential for enhanced generalization in real-world robotic applications.

Implications and Future Directions

The implications of this work are profound both theoretically and practically. Theoretically, it suggests a shift towards harnessing image generation capabilities of diffusion models for action-related tasks, aligning action prediction with visual generative models. Practically, this approach paves the way for developing more adaptable robotic systems that are capable of complex manipulation tasks without exhaustive scene-specific data provisioning.

Future research directions may include exploring the integration of diffusion models with reinforcement learning frameworks to enable the discovery of novel behaviors beyond behavior cloning. Additionally, enhancing the speed of diffusion processes would further augment the real-time applicability of such models in robotic settings. Considering safety and reliability aspects, particularly in tasks involving human interaction, remains a crucial aspect of ongoing exploration.

In summary, the paper demonstrates that the use of internet-pretrained diffusion models, fine-tuned to address action-generation, offers a compelling avenue for advancing visuomotor control, marking a significant leap in the interaction between image generation and robotic manipulation.