ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

Abstract: Chinese landscape painting is a gem of Chinese cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic Chinese landscape paintings. However, challenges such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating Chinese landscape painting videos. In this paper, we propose CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. The source code and dataset are available at https://anonymous.4open.science/r/ConCLVD-EFE3.

• The paper presents ConCLVD, a diffusion model that transforms static Chinese landscape images into dynamic, high-fidelity videos through trainable motion modules.
• It integrates a dual-attention mechanism, utilizing versatile and sparse-causal attention to ensure both global coherence and local temporal continuity.
• The authors developed the CLV-HD dataset and introduced contrastive learning and SOFP for effective frame interpolation and smoother video transitions.

ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

Introduction

The paper introduces ConCLVD, a novel approach for generating videos in the style of Chinese landscape paintings using text-to-video (T2V) diffusion models. Bridging traditional Chinese art with modern technology, this method employs the Stable Diffusion framework, augmented by a motion module and contrastive learning strategies, to transform static landscape imagery into dynamic video compositions.

Architecture Overview

ConCLVD integrates a trainable motion module and a noise adapter within a frozen Stable Diffusion framework to accommodate dynamic video generation. The design encapsulates a dual-attention mechanism—Versatile Attention and Sparse-Causal Attention—to effectively manage temporal data sequences. Versatile Attention connects every frame in a sequence globally, whereas Sparse-Causal Attention focuses on local causality, ensuring each frame only refers to its predecessor for continuity.

Figure 1: Overview of ConCLVD. Left: the architecture. ConCLVD integrates a trainable motion module based on a frozen Stable Diffusion and introduces a noise adapter to accommodate contrast learning of noise in latent space. Right: the inference framework.

Motion modules are strategically placed to facilitate the seamless transition between frames, capturing the dynamic elements inherent in Chinese landscape painting videos. This architecture maintains the artistic essence while modeling motion across frames.

Dataset Development

The authors introduced CLV-HD, a specialized dataset comprising 1,300 text-video pairs that reflect the stylized diversity of Chinese landscape paintings. This dataset addresses the previous lack of resources, enabling models to learn intrinsic art forms' unique characteristics through deep learning. Comprising both traditional and modern aesthetics, this dataset serves as a pivotal resource for advancing artistic video synthesis technologies.

Methodological Innovations

A prominent feature of ConCLVD is its incorporation of contrastive learning within the latent space. Consecutive noise vectors, corresponding to sequential video frames, are treated as positive pairs, while widely spaced frames form negative pairs, enhancing the model's ability to discern temporal nuances through unsupervised learning methods.

Figure 2: Design of Motion Module. The motion module is inserted following each image layer of the pre-trained SD to process video data.

Additionally, the introduction of Sparse Optical Flow Projection (SOFP) for frame interpolation effectively improves video smoothness. SOFP uses optical flow projections to interpolate between frames, allowing ConCLVD to produce coherent and fluid video sequences.

Figure 3: Detailed explanation of the attention mechanism. The above represents Versatile Attention, where each frame is related to every other frame; the below represents Sparse-Causal Attention, where each frame only focuses on its previous frame.

Experimental Validation

The experimental framework was deployed on an NVIDIA RTX 3090, showcasing ConCLVD's capability to produce high-definition video content with limited computational resources. The video outputs achieved high fidelity in terms of ink style representation and dynamic scene transitions consistent with traditional Chinese landscapes.

Quantitative comparisons against baseline models highlighted ConCLVD's superior performance on metrics such as temporal consistency and stylistic authenticity. Noteworthy is its ability to replicate subtle artistic nuances across video sequences, outperforming prior models in aesthetic preservation and text alignment.

Figure 4: Illustration of SOFP Interpolation. $V_{(t \rightarrow T)}$ denotes the optical flow used for interpolation between frames.

Conclusion

ConCLVD represents a significant advancement in the fusion of Chinese landscape art with digital video synthesis, enhancing cultural preservation through contemporary mediums. Its novel approach via diffusion models, augmented motion modules, and dynamic interpolation strategies, extend the boundaries of artistic video generation. The release of the CLV-HD dataset and the open-source nature of ConCLVD promises further exploration and development in the field, reinforcing its role as an invaluable tool for bridging traditional artistry with emerging technologies.