MiniMax-Remover: Taming Bad Noise Helps Video Object Removal

Abstract: Recent advances in video diffusion models have driven rapid progress in video editing techniques. However, video object removal, a critical subtask of video editing, remains challenging due to issues such as hallucinated objects and visual artifacts. Furthermore, existing methods often rely on computationally expensive sampling procedures and classifier-free guidance (CFG), resulting in slow inference. To address these limitations, we propose MiniMax-Remover, a novel two-stage video object removal approach. Motivated by the observation that text condition is not best suited for this task, we simplify the pretrained video generation model by removing textual input and cross-attention layers, resulting in a more lightweight and efficient model architecture in the first stage. In the second stage, we distilled our remover on successful videos produced by the stage-1 model and curated by human annotators, using a minimax optimization strategy to further improve editing quality and inference speed. Specifically, the inner maximization identifies adversarial input noise ("bad noise") that makes failure removals, while the outer minimization step trains the model to generate high-quality removal results even under such challenging conditions. As a result, our method achieves a state-of-the-art video object removal results with as few as 6 sampling steps and doesn't rely on CFG, significantly improving inference efficiency. Extensive experiments demonstrate the effectiveness and superiority of MiniMax-Remover compared to existing methods. Codes and Videos are available at: https://minimax-remover.github.io.

Overview of MiniMax-Remover: Taming Bad Noise Helps Video Object Removal

The paper introduces MiniMax-Remover, a novel two-stage approach for video object removal that addresses challenges inherent to video editing tasks such as hallucinated objects and visual artifacts. Existing methods often rely on computationally expensive techniques such as @@@@1@@@@ (CFG) and require numerous sampling steps, which impede inference efficiency. MiniMax-Remover innovates by simplifying the architecture and applying a minimax optimization strategy to improve both quality and speed.

Two-Stage Framework

The proposed method employs a two-stage training framework:

1. Stage 1 - Simplified Video Generation Model: The first stage modifies a pretrained video generation model by removing textual inputs and cross-attention layers, resulting in a more lightweight and efficient architecture. This adjustment is motivated by the observation that text conditions do not best suit the task of video object removal. Instead, learnable contrastive condition tokens are utilized to control the inpainting process while integrating directly into the self-attention mechanism of the model. This leads to a more streamlined architecture with reduced computational demands.

2. Stage 2 - Minimax Optimization for Robustness:

The second stage leverages minimax optimization strategies to train the model on adversarial input noise, referred to as "bad noise". An inner maximization identifies these challenging conditions that provoke model failures, and an outer minimization step trains the model to generate high-quality removal results under such adversarial circumstances. This approach eliminates the dependency on CFG and achieves state-of-the-art results with only six sampling steps, drastically improving inference speed.

Experimental Results

The effectiveness of the MiniMax-Remover is validated through comprehensive experiments, demonstrating superior results compared to existing methods:

• Efficiency: Achieving video object removal with only 6 sampling steps compared to the usual 50 steps or more required by other techniques.
• Quality and Robustness: Outpacing previous methods in terms of visual quality metrics such as SSIM and PSNR, and maintaining temporal consistency, even under adversarial noise conditions.
• Practical Applicability: Better performance in terms of both inference speed and visual fidelity, making it a viable solution for real-world applications.

Implications and Speculations for Future Developments

The research presented in this paper holds significant implications for the field of AI-driven video editing:

• Practical Applications: The reduction in computational overhead makes this approach attractive for practical deployments in video editing software or online platforms where efficiency is critical.
• Theoretical Insights: The minimax optimization not only enhances robustness but also pushes the boundaries of video object removal capabilities without dependency on CFG, offering new directions for future research in diffusion models and adversarial training.
• Future Developments: Further exploration of larger datasets for training and the possible integration of smaller, more efficient VAE components could yield improvements in both speed and performance, making it feasible for a broader range of applications.

The paper successfully demonstrates that intelligently managing adversarial noise in the video diffusion context can lead to substantial advancements in both efficiency and capability for video object removal tasks, paving the way for practical and theoretical advancements in the domain of AI video editing.