Hierarchical Dynamic Image Harmonization (2211.08639v3)

Abstract: Image harmonization is a critical task in computer vision, which aims to adjust the foreground to make it compatible with the background. Recent works mainly focus on using global transformations (i.e., normalization and color curve rendering) to achieve visual consistency. However, these models ignore local visual consistency and their huge model sizes limit their harmonization ability on edge devices. In this paper, we propose a hierarchical dynamic network (HDNet) to adapt features from local to global view for better feature transformation in efficient image harmonization. Inspired by the success of various dynamic models, local dynamic (LD) module and mask-aware global dynamic (MGD) module are proposed in this paper. Specifically, LD matches local representations between the foreground and background regions based on semantic similarities, then adaptively adjust every foreground local representation according to the appearance of its $K$-nearest neighbor background regions. In this way, LD can produce more realistic images at a more fine-grained level, and simultaneously enjoy the characteristic of semantic alignment. The MGD effectively applies distinct convolution to the foreground and background, learning the representations of foreground and background regions as well as their correlations to the global harmonization, facilitating local visual consistency for the images much more efficiently. Experimental results demonstrate that the proposed HDNet significantly reduces the total model parameters by more than 80\% compared to previous methods, while still attaining state-of-the-art performance on the popular iHarmony4 dataset. Notably, the HDNet achieves a 4\% improvement in PSNR and a 19\% reduction in MSE compared to the prior state-of-the-art methods.

• The paper introduces HDNet, a novel approach that dynamically harmonizes composite images by combining adaptive local and global adjustment modules.
• The methodology utilizes a Local Dynamic module to align foreground features with background semantics and a Mask-aware Global Dynamic module for seamless visual integration.
• Empirical results show HDNet achieves state-of-the-art performance on the iHarmony4 dataset with an over 80% reduction in model parameters, enabling deployment on edge devices.

Hierarchical Dynamic Image Harmonization

The paper "Hierarchical Dynamic Image Harmonization" by Haoxing Chen et al. presents an innovative approach towards solving the intricate problem of image harmonization, a core task in the domain of computer vision. Image harmonization is essential for integrating disparate image patches into a seamless, realistic image by adjusting the composite's foreground to match its background. Traditional methods focused on low-level hand-crafted appearance statistics have proven inadequate for complex scenes, paving the way for this research which introduces a robust solution termed as Hierarchical Dynamic Network (HDNet).

Overview and Methodology

The proposed HDNet aims to enhance image harmonization by dynamically adjusting the feature representation from a local to a global perspective. This process is achieved through the development of two novel modules: the Local Dynamic (LD) module and the Mask-aware Global Dynamic (MGD) module.

1. Local Dynamic Module: The LD module addresses the limitations of global feature transformation by adapting to local regions based on semantic similarities. For each local representation in the foreground, the LD module identifies the $K$-nearest neighbors from the background and uses these neighbors to reconstruct the foreground representation. This adaptive approach ensures finer-level adjustments and semantic alignment between foreground and background.
2. Mask-aware Global Dynamic Module: The MGD module learns representations for both foreground and background, enhancing global harmonization by addressing local visual inconsistencies. It applies distinct convolutional filters to seamlessly adapt to the variations in different image regions, thereby facilitating efficient and coherent image harmonization.

Experimental Results

The empirical evaluation of HDNet showcases its significant performance advantages over existing methods. The research communicates a reduction in model parameters by over 80% while achieving state-of-the-art results on the iHarmony4 dataset. Additionally, the paper introduces HDNet-lite, a lightweight model with only 0.65MB of parameters, which demonstrates competitive performance.

The paper quantitatively compares HDNet against state-of-the-art methods including RainNet, DoveNet, and Harmonizer, revealing HDNet's superior performance concerning mean square error (MSE) and peak signal-to-noise ratio (PSNR) across multiple sub-datasets. The authors provide a rigorous ablation paper, confirming the efficacy of each component within HDNet.

Implications and Future Work

The implications of this research are both practical and theoretical. Practically, the reduced model size and high efficiency make HDNet suitable for deployment on edge devices, enabling real-time application in mobile scenarios. Theoretically, HDNet opens new avenues for exploring hierarchical dynamics in other computer vision tasks beyond image harmonization.

Future directions for research as suggested by the authors involve addressing the challenge of mask dependency, as performance is currently contingent on the availability of a reliable mask. Exploring unsupervised methods for mask generation or integrating HDNet with generative models could potentially mitigate this limitation.

In conclusion, the development of HDNet marks a significant advancement in image harmonization, leveraging hierarchical dynamic adaptation to achieve superior performance while maintaining efficiency. This work thus constitutes a notable contribution to the field of computer vision, particularly in tasks requiring the seamless integration of disparate visual elements.