Composite Generative Adversarial Networks (CGAN)

• Composite Generative Adversarial Networks (CGANs) are architectures that decompose image synthesis into parts using multiple generators to create coordinated outputs.
• They utilize techniques such as alpha compositing and multi-head self-attention to enhance interpretability, disentanglement, and controlled generation.
• Empirical evaluations show that CGANs improve unsupervised segmentation, class-conditional synthesis, and medical imaging results with robust performance metrics.

Composite Generative Adversarial Networks (CGANs) represent a class of architectures and inductive biases for generative image modeling that explicitly encode compositional structure—either by part, by object, or by condition—in the generative process. These frameworks leverage one or more generators (and potentially discriminators) to synthesize images or scenes by decomposing complexity into coordinated subcomponents, addressing challenges of interpretability, controllability, disentanglement, and stability that conventional GANs often face.

1. Foundational Architectures and Compositionality

Initial proposals for Composite Generative Adversarial Networks replace the canonical single-generator paradigm with a cascade or ensemble of generators. In the architecture described by Kwak and Zhang, a CGAN comprises $n$ generators $G_1, ..., G_n$, each responsible for producing an RGBA image "part" (e.g., background, face, hair), with these partial results combined via a differentiable alpha compositing process to form the final image output. Each generator receives its own latent code, potentially further contextualized through sequential dependencies modeled by an RNN (such as an LSTM cell), enforcing consistent part-to-whole assembly (Kwak et al., 2016). Mathematically, at each stage, the composite is updated:

$$O^{(t)}_{ij,RGB} = O^{(t-1)}_{ij,RGB}\bigl(1 - C^{(t)}_{ij,A}\bigr) + C^{(t)}_{ij,RGB}\,C^{(t)}_{ij,A}$$

where $C^{(t)}$ is the RGBA tensor output of $G_t$.

A divergent but related compositional principle is explored by Eslami et al., who demonstrate that object-centric generation can be realized by weight-tying a set of $K$ generators (each with independent latent codes $z_i$), composing scene images by summing the outputs and optionally extending to multi-head self-attention layers for modeling dependencies and occlusions. Here, the generator produces:

$x = \sum_{i=1}^K G(z_i)$

with extensions including an explicit background generator and front-to-back alpha-compositing (Steenkiste et al., 2018).

2. Compositional Conditional GANs: Class and Attribute Conditioning

Most contemporary cGANs—Conditional GANs—handle composition in the class or attribute sense, generating $x \sim p_{data}(x|y)$ conditioned on discrete or continuous labels $y$. These frameworks encode $y$ into the generator and/or discriminator in various ways. In City-GAN, the generator receives a concatenation of a random noise vector $z$ and a one-hot city label $c$ at its input layer, while the discriminator is made city-aware by concatenating the label map channel-wise to every input pixel, enabling convolutional layers to be conditioned at all spatial locations (Bachl et al., 2019). This pixelwise label-injection is more effective than late-stage conditioning, yielding sharper, more distinct samples.

Unified frameworks such as ECGAN leverage joint probability decompositions to bring together approaches like Auxiliary Classifier GAN (ACGAN), Projection GAN (ProjGAN), and ContraGAN—architectures that variously employ classifiers, projection-based discriminators, and contrastive objectives to enforce conditional generation (Chen et al., 2021). The ECGAN framework parameterizes the joint distribution $p_\theta(x, y)$ via a neural network $f_\theta(x)[y]$, yielding conditional and classification losses, and supporting both energy-model and adversarial training.

3. Object- and Part-level Composition: Inductive Biases and Extensions

Object compositionality introduces explicit modeling of scenes as assemblies of independent or interacting entities. In "Investigating Object Compositionality in GANs," two core extensions are delineated:

1. Relational Stage: Latent codes $\{z_i\}$ for each object are refined by multi-head self-attention, allowing inter-object dependencies to be modeled via attention weightings over all other objects' latent spaces.
2. Alpha-based Occlusion Modeling: With RGBA outputs for each generator, a fixed order compositing mechanism recursively blends object layers and background, facilitating occlusion, transparency, and scene hierarchies (Steenkiste et al., 2018).

These inductive biases drive unsupervised instance segmentation: after CGAN training, decomposed alpha masks can serve as pseudo-ground truth for training a segmentation network, achieving Adjusted Rand Index (ARI) values on par with supervised models in settings such as Multi-MNIST and CLEVR.

4. Self-consistent Composition and Decomposition Approaches

Compositional GANs also address explicit multi-entity compositing with self-consistency and reversibility constraints. The CoDe (Composition-by-Decomposition) network learns to synthesize a composite image from two inputs originating from distinct domains (e.g., a chair and a table) via a composition generator and PatchGAN discriminator, followed by an explicit decomposition network that predicts the original inputs and soft pixelwise masks. Loss functions combine adversarial, L1 reconstruction, and cross-entropy masking objectives, enforcing that $G_{dec}(G_c(x, y)) \approx (x, y)$. This cycle-like structure, augmented by spatial transformer and appearance flow modules for geometric alignment, enables the model to learn relative scaling, spatial layout, and occlusion automatically (Azadi et al., 2018).

5. Training Dynamics, Regularization, and Stability Mechanisms

Composite and conditional architectures often require architectural and regularization mechanisms to achieve stable and interpretable training:

• Alpha regularization, as in (Kwak et al., 2016), ensures that roles among generators are distributed—penalizing domination of opacity by a single stage.
• Dual discriminator setups, as deployed in MSGDD-cGAN, separate the enforcement of input information retention from output realism. Multi-scale gradient flows are established by connecting both encoder and decoder feature maps at several resolutions to their respective discriminators. Generator and discriminator losses are constructed in the least-squares GAN style, with multiple L1 losses applied at every relevant scale (Naderi et al., 2021).
• Architectural choices, such as injection of conditioning information at every spatial position, multi-scale L1 objectives, and the use of U-Net backbones, all contribute to mode-collapse avoidance, balance between faithfulness to input and realism of outputs, and resilience to hyperparameter perturbations (Naderi et al., 2021, Bachl et al., 2019).

6. Applications and Empirical Evaluations

Composite GAN architectures have been evaluated in a range of synthetic and real domains:

• Hierarchical part synthesis: Kwak and Zhang show qualitatively that with $n=3$ generators, backgrounds, faces, and details are synthesized in stages across CelebA and Oxford Flowers datasets (Kwak et al., 2016).
• Unsupervised segmentation: Eslami et al. demonstrate that ARI for segmentation using unsupervised CGAN-generated masks is nearly identical to ground-truth-trained baselines (Steenkiste et al., 2018).
• Class-conditional synthesis: City-GAN qualitatively establishes sharp separation between city-specific architectural styles by label manipulations in the latent code space (Bachl et al., 2019).
• Paired and unpaired conditional composition: In CoDe, qualitative and AMT user studies confirm the model’s ability to capture spatial relationships, occlusions, and scale between independently sampled object domains (Azadi et al., 2018).
• Medical image segmentation: MSGDD-cGAN achieves a +3.18% F1 score improvement over pix2pix baselines on fetal ultrasound head segmentation, evidencing the effective balancing of condition fidelity and output distribution fit (Naderi et al., 2021).
• Large-scale conditional synthesis: ECGAN delivers lower FID and higher Inception Scores on CIFAR-10, TinyImageNet, and ImageNet compared with prior conditional GANs, substantiating the benefits of joint distribution decomposition and classifier guidance (Chen et al., 2021).

7. Limitations, Open Questions, and Future Directions

Composite generative frameworks present certain unresolved challenges:

• The semantic richness and ordering of part-level generators remain emergent properties—roles may be unstable or generators may become degenerate unless regularized (Kwak et al., 2016).
• Scaling to high resolution necessitates sophisticated blending and mask prediction, as naive architectures struggle with sharp compositional boundaries.
• Geometric and appearance transformations are viable only for a limited number of rigid objects; current composition models have not fully addressed illumination, shadow, and fine-grained scene physics (Azadi et al., 2018).
• The general applicability to non-image modalities and the integration with structured attention or explicit scene graphs remain open questions.

Empirical validation of composite architectures has mostly centered on visual and qualitative metrics; robust quantitative evaluation, generalization to diverse data types, and further synergistic integration of compositional principles are active areas of research.