VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence (2312.02087v2)

Abstract: Current diffusion-based video editing primarily focuses on structure-preserved editing by utilizing various dense correspondences to ensure temporal consistency and motion alignment. However, these approaches are often ineffective when the target edit involves a shape change. To embark on video editing with shape change, we explore customized video subject swapping in this work, where we aim to replace the main subject in a source video with a target subject having a distinct identity and potentially different shape. In contrast to previous methods that rely on dense correspondences, we introduce the VideoSwap framework that exploits semantic point correspondences, inspired by our observation that only a small number of semantic points are necessary to align the subject's motion trajectory and modify its shape. We also introduce various user-point interactions (\eg, removing points and dragging points) to address various semantic point correspondence. Extensive experiments demonstrate state-of-the-art video subject swapping results across a variety of real-world videos.

• The paper presents a novel VideoSwap framework that leverages interactive semantic point correspondence for effective subject swapping in videos.
• It employs a latent diffusion model with motion layers to ensure motion alignment and background consistency even when the subject shape changes.
• Human evaluations demonstrate that VideoSwap outperforms state-of-the-art methods in preserving subject identity and maintaining edit quality.

In the ever-expanding field of video editing, a new framework called VideoSwap has been introduced, unlocking the potential for more personalized and interactive subject swapping in videos. This framework allows users to replace the main subject of a source video with a different subject while retaining the original motion trajectory and background consistency.

Prior approaches often struggled with subject swapping when the target needed to undergo a shape change. VideoSwap tackles this by recognizing that adjusting a small number of key 'semantic points' can effectively describe a subject's motion and consequently change the subject's shape within the video. Semantic points refer to specific locations on the subject that are meaningful for tracking its motion—for example, the tail and wings of an airplane.

VideoSwap operates by first extracting these semantic points’ trajectories and their corresponding embeddings from the source video. Then, leveraging user interactions, such as adding or removing points or even dragging points to modify the subject's shape on keyframes, the system adapts these semantic points to guide the editing process, offering a significant leap in customized video editing capabilities.

The system is developed upon a Latent Diffusion Model with added motion layers that ensure essential temporal consistency, crucial for maintaining fluidity and realism in video editing. Semantic points offer correspondence that can transfer motion while accommodating shape changes. Users have the option to adjust them to better fit the new subject's shape, especially useful in cases where the source and target subjects are not identical in form.

The results of VideoSwap have been demonstrably superior, offering significant improvements in subject identity preservation, motion alignment, and overall video edit quality compared to state-of-the-art methods. Human evaluations have reaffirmed these results, showing a clear preference for the videos edited with VideoSwap.

Despite its impressive capabilities, VideoSwap has some limitations, such as potential inaccuracies in semantic point tracking due to self-occlusions or significant viewpoint changes, as well as challenges in real-time interactive editing due to preprocessing times. Moreover, while the tool primarily aims to offer creative and customized video editing options, like all powerful technologies, there is a potential for misuse. Approaches like subtle watermarking or sound model customization hygiene are critical discussions for the community, aiming to ensure that such technology is utilized positively and ethically.

In conclusion, VideoSwap represents a step forward for those looking to create customized video content. Whether for professional editors seeking to refine the storytelling in their video productions or for individuals wanting to inject personalized elements into their footage, VideoSwap opens up new possibilities for video editing that were previously difficult to achieve. With continued development and ethical considerations, tools like VideoSwap could redefine the standard for personalized video editing in the creative industry.