Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data (1802.10151v2)

Abstract: Learning inter-domain mappings from unpaired data can improve performance in structured prediction tasks, such as image segmentation, by reducing the need for paired data. CycleGAN was recently proposed for this problem, but critically assumes the underlying inter-domain mapping is approximately deterministic and one-to-one. This assumption renders the model ineffective for tasks requiring flexible, many-to-many mappings. We propose a new model, called Augmented CycleGAN, which learns many-to-many mappings between domains. We examine Augmented CycleGAN qualitatively and quantitatively on several image datasets.

• The paper introduces novel stochastic mappings incorporating auxiliary latent variables to capture one-to-many relationships in image translation.
• It employs cycle and marginal matching losses to stabilize optimization and produce diverse outputs from unpaired data.
• The model demonstrates improved performance in semi-supervised settings and on tasks like edge-to-photo translation with superior quantitative metrics.

Augmented CycleGAN: Many-to-Many Mappings from Unpaired Data

The paper, "Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data," proposes an innovative extension to the existing CycleGAN framework, which is limited by its assumption of approximately deterministic and one-to-one mappings. This research aims to address these limitations by introducing the Augmented CycleGAN model, capable of learning many-to-many mappings between domains, thereby enhancing its application potential in tasks involving complex relationships, such as image-to-image translation where paired data may be sparse or unavailable.

Model Overview

Augmented CycleGAN extends the conventional CycleGAN by augmenting domains with auxiliary latent variables, permitting the modeling of one-to-many relationships within image-to-image translation tasks. This is achieved by introducing stochastic mappings between domains, where each mapping additionally considers a noise vector, allowing for the generation of diverse outputs from a single input. The use of latent spaces augments the original CycleGAN domain pairs, leading to mappings defined as $A \times Z$ to $B \times Z$ and vice versa, where $Z$ represents the latent space.

Core Contributions

1. Stochastic Mappings with Latent Variables: The paper's primary contribution is the development of mappings that incorporate auxiliary latent variables, allowing the generator to capture a broader distribution and produce multiple plausible outputs for each input. This effectively addresses the deterministic constraint of the original CycleGAN.
2. Cycle and Marginal Matching Losses: Augmented CycleGAN utilizes both cycle-consistency and marginal matching losses to stabilize the network optimization. Cycle-consistency is traditionally incorporated in CycleGAN, ensuring that transformations between domains are reversible. Here, this is extended to augmented spaces, allowing the model to retain rich diversity through stochastic elements.
3. Semi-Supervised Learning Capability: The model enhances its versatility by enabling semi-supervised learning when partial pairing information is available, using real pairs along with the unpaired data, further regularizing the mapping functions.

Empirical Evaluation

Evaluations were conducted on domains with high diversity, such as edges-to-shoes and male-to-female face mappings. The proposed model demonstrated its capacity to generate multiple diverse outputs from a single input in scenarios where the original CycleGAN might produce repetitive or less varied results due to its deterministic nature.

The model's performance was quantitatively assessed using metrics such as L1 loss and Mean Squared Error (MSE) on tasks involving edge-to-photo translations, with superior performance over conventional and even semi-supervised variants of $\Delta$-GAN models.

Implications and Future Work

Augmented CycleGAN represents a significant step towards addressing the limitations of deterministic mapping functions by incorporating stochastic processes and latent variable architectures, encouraging broader applications across various tasks with unpaired data constraints. The model's ability to handle many-to-many mappings opens up new directions for research in applications that require rich, multimodal outputs, such as advanced image editing and data augmentation for training deep learning models.

Future work may explore further optimization strategies for cycle-consistency in stochastic settings, investigate the integration with other conditional generative models like VAEs, and extend the framework to accommodate even more complex domain transformations beyond pairs, potentially utilizing additional forms of implicit or explicit domain knowledge.