Instance Normalization: The Missing Ingredient for Fast Stylization

Abstract: It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at https://github.com/DmitryUlyanov/texture_nets. Full paper can be found at arXiv:1701.02096.

• The paper demonstrates that replacing batch normalization with instance normalization yields higher quality stylized images in a feed-forward network.
• It shows that instance normalization preserves contrast and details by normalizing each image instance separately, reducing artifacts.
• Empirical results confirm the method improves both efficiency and quality, making real-time image stylization more practical.

Instance Normalization: The Missing Ingredient for Fast Stylization

The paper “Instance Normalization: The Missing Ingredient for Fast Stylization” authored by Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, presents a significant enhancement in the domain of feed-forward neural networks used for real-time image stylization. The fundamental contribution lies in replacing batch normalization with instance normalization, a change that yields substantial improvements in the quality of stylized images.

Context and Background

The method introduced by Gatys et al. marked a pivotal advance in the field of style transfer by leveraging deep convolutional networks to match statistics from style and content images. However, this approach suffers from high computational costs as it relies on iterative optimization, rendering it impractical for real-time applications.

To address this, Ulyanov et al. and Johnson et al. previously developed feed-forward networks that generate stylized images in a single pass. Despite promising results, these methods could not match the quality achieved by the slower optimization-based techniques of Gatys et al. The inefficiency in capturing detailed style elements motivated the authors to refine these networks further.

Methodology

The innovation presented in this paper revolves around a simple yet effective architectural modification: replacing batch normalization layers with instance normalization layers in the generator networks used for image stylization. Batch normalization normalizes the input across the batch, which can introduce undesirable artifacts from batch statistics. Conversely, instance normalization normalizes each instance separately, thus preserving instance-specific details and contrast better.

Formally, if $x \in \mathbb{R}^{T \times C \times W \times H}$ is an input tensor, instance normalization can be described as follows:

$$y_{tijk} = \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}}, \quad \mu_{ti} = \frac{1}{HW} \sum_{l=1}^W \sum_{m=1}^H x_{tilm}, \quad \sigma_{ti}^2 = \frac{1}{HW} \sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - \mu_{ti})^2$$

This replaces the mean and variance calculations in batch normalization with those calculated for each instance independently, avoiding the influence of inter-batch variations.

Experimental Validation

The experiments demonstrate the efficacy of the proposed modification by comparing the performance of generator networks using batch normalization versus instance normalization. The results indicate that instance normalization significantly enhances the quality of generated images with fewer artifacts, aligning closer to the results produced by the optimization-based methods of Gatys et al.

Two architectures were tested: those used in Ulyanov et al. and Johnson et al. Despite their structural differences, both architectures benefitted from this modification. Qualitative comparisons show that the instance-normalized networks outperformed their batch-normalized counterparts, particularly in preserving the contrast and details of the stylized images.

Practical and Theoretical Implications

This enhancement in image stylization has broad implications for practical applications requiring real-time performance, such as video processing, photo editing software, and augmented reality. The reduction in computational overhead without sacrificing quality aligns with the growing demand for efficiency in practical deployments of style transfer models.

Theoretically, the success of instance normalization points to its potential benefits in other domains of image generation and transformation. The paper opens avenues for exploring instance normalization in tasks beyond style transfer, such as super-resolution, image generation, and even image discrimination tasks.

Future Directions

Given the promising results, several future research directions present themselves. First, further exploration into the combination of instance normalization with other normalization techniques could yield even more robust architectures. Second, extending the principles found in this work to video stylization may help mitigate temporal inconsistencies that plague current methods. Lastly, applying instance normalization to other machine learning models could refine performance across various tasks, suggesting a broader potential impact.

Conclusion

The substitution of batch normalization with instance normalization in feed-forward networks for image stylization represents a significant stride in generating high-quality stylized images efficiently. This modification simplifies the learning process and stabilizes the generated results, showcasing a methodological improvement with far-reaching implications for real-time applications and future research in neural network architectures.