Adversarial Image Composition with Auxiliary Illumination

Abstract: Dealing with the inconsistency between a foreground object and a background image is a challenging task in high-fidelity image composition. State-of-the-art methods strive to harmonize the composed image by adapting the style of foreground objects to be compatible with the background image, whereas the potential shadow of foreground objects within the composed image which is critical to the composition realism is largely neglected. In this paper, we propose an Adversarial Image Composition Net (AIC-Net) that achieves realistic image composition by considering potential shadows that the foreground object projects in the composed image. A novel branched generation mechanism is proposed, which disentangles the generation of shadows and the transfer of foreground styles for optimal accomplishment of the two tasks simultaneously. A differentiable spatial transformation module is designed which bridges the local harmonization and the global harmonization to achieve their joint optimization effectively. Extensive experiments on pedestrian and car composition tasks show that the proposed AIC-Net achieves superior composition performance qualitatively and quantitatively.

• The paper introduces AIC-Net, a dual-branch GAN that simultaneously synthesizes realistic shadows and harmonizes image textures during composition.
• It employs an auxiliary illumination model and a guided feature filter to independently optimize shadow generation and style transfer.
• Experimental results with FID and AMT scores demonstrate significant improvements in image realism and seamless integration.

Adversarial Image Composition with Auxiliary Illumination

The paper "Adversarial Image Composition with Auxiliary Illumination" introduces the Adversarial Image Composition Net (AIC-Net). This method addresses the challenges of image composition by integrating foreground objects into background images while realistically generating shadows, thereby enhancing the perceived realism.

AIC-Net Architecture

Integrating Shadow and Style Transfer

AIC-Net features a dual-branch generation strategy to separately handle style transfer and shadow generation, thus optimizing both tasks and overcoming their conflicting objectives. This approach relies on the concurrent application of a shadow branch and a texture branch.

• Shadow Branch: The shadow branch synthesizes shadows based on the illumination direction, which is derived using an auxiliary illumination model. This model infers illumination conditions from the background image, translating them into shadow generation parameters using spherical harmonics (SH).
• Texture Branch: This branch focuses on harmonizing the texture of the foreground objects with that of the background. A guided feature filter ensures that while style is applied, the content features of the object remain intact.

  Figure 1: The structure of the proposed AIC-Net, illustrating the dual branches for shadow generation and style transfer.

Spatial Transformer Module (STM)

The STM bridges local and global harmonization, ensuring end-to-end differentiability. It estimates a homography matrix to effectively align the transformed image region, facilitating seamless integration of foreground appendages with the broader image.

Figure 2: Global harmonization using a spatial transformation matrix to achieve coherence between the local and global context.

Generative Adversarial Framework

AIC-Net leverages a GAN-based structure for adversarial training. This involves two discriminators:

• Local Discriminator: It ensures that neither style transfer nor shadow generation degrades the local image quality.
• Global Discriminator: This discriminator aids in mitigating boundary artifacts, ensuring global image consistency.

Experimental Validation

Quantitative Analysis

The performance of AIC-Net is quantitatively evaluated using metrics such as FID and AMT scores, demonstrating superior harmonization quality compared to existing methods. The effective disentangling of shadow from style transfer and the robust lighting estimation contributed to significant improvements in image realism.

Figure 3: Comparative analysis of local and global harmonizations by AIC-Net versus other methods, highlighting its efficacy in producing seamless compositions.

Qualitative Analysis

Figures in the study reveal AIC-Net’s proficiency in simulating true-to-life shadows across diverse environments and conditions, validating the model’s adaptability to varying illumination scenarios.

Figure 4: AIC-Net’s ability to adaptively generate shadows under various illumination and weather conditions.

Conclusion

AIC-Net significantly enhances image harmonization through realistic shadow generation and effective style transfer, facilitated by its dual-branch architecture and illumination model. The proposed method demonstrates promise for applications needing high-fidelity image synthesis and augmented reality integration, positioning itself as a noteworthy advancement in adversarial image composition methodologies. Future work may explore further extensions for more complex environmental lighting scenarios.