Conditional Generation: Models & Applications

• Conditional generation is a technique that produces data by modeling the conditional probability distribution p(x|c), enabling controlled synthesis with explicit signals.
• It integrates diverse deep generative architectures such as autoregressive models, cGANs, cVAEs, normalizing flows, and diffusion models to handle various structured data types.
• Applications include controlled image synthesis, text infilling, and 3D model manipulation, with evaluation metrics like FID, LPIPS, and classifier accuracy ensuring performance quality.

Conditional generation refers to the process of sampling from a conditional probability distribution over structured data—such as images, text, or point clouds—given side-information or context variables. This paradigm enables controllable synthesis, manipulation, and completion of data, given explicit conditioning signals such as class labels, multimodal observations, or attribute vectors. Conditional generation is realized across multiple deep generative model architectures, including autoregressive models, variational autoencoders, normalizing flows, generative adversarial networks, diffusion models, and probabilistic graphical models. The following sections provide a comprehensive overview rooted in the design principles, technical frameworks, and recent methodological advances defining conditional generation.

1. Mathematical Formulation and Model Classes

Let $x$ denote the structured data (e.g., image, text sequence), and $c$ denote the conditioning variable (e.g., class label, attribute vector, spatial mask, or another modality). The core objective is to model $p_\theta(x \mid c)$, the conditional data distribution, such that the generator can sample diverse, high-fidelity outputs consistent with $c$.

Common conditional generation models and representative formalisms:

• Autoregressive models (e.g., Conditional PixelCNN):

$$p(x \mid c) = \prod_{i=1}^{n} p(x_i \mid x_{1:i-1}, c)$$

Conditioning enters as additive or multiplicative biases per layer (Oord et al., 2016).

• Conditional GANs (cGANs):

Both generator $G$ and discriminator $D$ are conditioned on $c$. The training objective is:

$$\min_G \max_D \ \mathbb{E}_{x,c}[\log D(x, c)] + \mathbb{E}_{z,c}[\log(1 - D(G(z, c), c))]$$

(Sagong et al., 2019, Mahfouz et al., 2022).

• Conditional VAEs (cVAEs):

Incorporate $c$ both in encoder and decoder. For latent $c$0:

$c$1

The evidence lower bound (ELBO) incorporates $c$2 (Harvey et al., 2021, Duan et al., 2019).

• Conditional Flows:

The flow $c$3 is parameterized as $c$4, with invertibility and tractable Jacobian, enabling $c$5 and direct attribute conditioning (Wielopolski et al., 2021, Das et al., 2021).

• Conditional Diffusion Models:

The denoising network $c$6 estimates the noise given $c$7, enabling conditional sampling or guidance (Graikos et al., 2023, Li et al., 2024).

• Probabilistic graphical models:

Conditional Probability Tables (CPTs) encode $c$8, extended to handle soft evidence and generalized conditioning (Garn et al., 2015).

The design of conditional generative models often involves integrating $c$9 into network architectures, loss functions, sampling algorithms, or post-hoc transformation modules.

2. Conditioning Mechanisms and Architectural Strategies

The technical approach to infusing conditioning signals into generative models is domain- and architecture-dependent:

Autoregressive image models:

• PixelCNN incorporates $p_\theta(x \mid c)$0 via additive projections at every gated convolutional layer. For global (location-independent) conditioning, a matrix projects $p_\theta(x \mid c)$1 into per-channel biases; for spatially-structured $p_\theta(x \mid c)$2, a feature map is supplied via a separate embedding network (Oord et al., 2016).

GANs:

• Early cGANs simply concatenate $p_\theta(x \mid c)$3 to the noise vector and input of $p_\theta(x \mid c)$4.
• Conditional Convolution (cConv) modulates the convolutional weights with per-class scaling and shifting, allowing for direct class-specific feature extraction per layer (Sagong et al., 2019).
• One-Vs-All (OVA) discriminators replace binary real/fake discrimination with multi-class output, separating real-vs-fake for each class to decouple gradients and improve stability (Xu et al., 2020).

VAEs and plugin approaches:

• Pre-trained unconditional VAEs can be “conditioned” by learning an amortized partial encoder that maps $p_\theta(x \mid c)$5 (e.g., observed pixels in inpainting) to a distribution over the unconditional VAE’s latent space. The decoder remains fixed (Harvey et al., 2021, Duan et al., 2019).
• Inference networks (e.g., I in (Chan et al., 2018)) can be trained to map auxiliary noise to posterior distributions $p_\theta(x \mid c)$6, supporting direct feed-forward conditional sampling.

Normalizing Flows and Plug-In Flows:

• Post-hoc flows (Flow Plugin Networks, FPNs) are attached between prior and latent space of a frozen generator/autoencoder, transform base noise to class/attribute-specific latent codes, enabling conditional generation, editing, and classification without retraining the base model (Wielopolski et al., 2021, Das et al., 2021).

Diffusion models:

• Conditional diffusion models are realized via (a) conditioning the noise estimation network, (b) classifier guidance operating on internal features or outputs, or (c) learning joint noisy trajectories for multiple modalities (e.g., image-depth pairs), supporting both direct conditioning and multi-signal fusion (Graikos et al., 2023, Li et al., 2024).

Flow Matching and PDE-Inductive Bias:

• EFM learns a “matrix field” $p_\theta(x \mid c)$7 via the generalized continuity equation, allowing the generative flow to be continuous in both time and conditional variable and penalizing pathological sensitivity in $p_\theta(x \mid c)$8 with Dirichlet energy minimization (Isobe et al., 2024).

3. Training Objectives and Optimization

The loss function and training procedure are tightly coupled to the generative framework:

Model Class	Primary Objective	Conditional Mechanism
Autoregressive	NLL (maximum likelihood)	Add/proj $p_\theta(x \mid c)$9 at each step
GAN/cGAN	Adversarial min–max	$c$0 input at $c$1, $c$2; OVA, cConv
VAE/cVAE	ELBO: recon + KL	$c$3 in encoder/decoder, plugin
Flow-based	NLL via change of variables	Conditional flows, plugin
Diffusion	MSE on noise, plus classifier guidance	$c$4, guidance
Bayesian/cPT	Fit to observational soft evidence	Regression, simplex mapping
EFM	Matrix-field regression on path stats	Dirichlet energy, MMOT coupling

Additional regularizers address issues such as mode collapse (cycle-consistency, MSE terms), diversity (diversity regularizers), disentanglement (auxiliary classifiers), and inductive bias (energy minimization).

4. Empirical Results, Modeling Trade-offs, and Limitations

Key experimental findings and trade-offs across domains:

• Image synthesis: Conditional GANs with OVA discrimination achieve faster and more stable convergence than ACGANs; cConv improves sample quality and class distinction (Sagong et al., 2019, Xu et al., 2020).
• Unsupervised conditional clustering: Double cycle-consistent GANs outperform ClusterGAN in both clustering accuracy and sample diversity across MNIST, Fashion-MNIST, and CIFAR-10 (Ding et al., 2019).
• Few-shot and plug-in conditionalization: FPNs enable zero-shot conditionalization on pre-trained models for attribute editing/classification; empirical classification accuracy for FPNs on MNIST exceeds 96% (Wielopolski et al., 2021).
• Probabilistic modeling: Regression-based CPT generation exceeds SME-elicited tables on diagnostic accuracy in effects-based decision tasks, with “soft evidence” handled natively (Garn et al., 2015).
• Text generation: Pre-train/plug-in architectures decouple modeling and condition adaptation, enabling rapid support of new conditions with competitive accuracy and diversity (Duan et al., 2019).
• Diffusion models: Guidance via denoiser representations (internal h_t) achieves competitive FID and semantic alignment in both attribute- and mask-conditioned synthesis with minimal labeled data (Graikos et al., 2023).
• Unified conditional diffusion (UniCon): A single model supports joint, conditional, inverse, coarse, and multi-signal synthesis at ~15% parameter overhead, outperforming specialized ControlNet/Readout methods on FID and task-specific metrics (Li et al., 2024).
• Inductive structure: Extended Flow Matching (EFM) provides continuity in both $c$5 and $c$6, leading to smoother interpolation in style transfer and lower Wasserstein error off-training conditions compared to black-box/diffusion-guided approaches (Isobe et al., 2024).

Significant limitations:

• GAN/cGAN approaches can suffer from mode collapse and insufficient coverage of multimodal uncertainty, partially mitigated by cycle-consistency and variational augmentation (Ding et al., 2019, Hu et al., 2019).
• Conditional sampling is ultimately restricted by the expressiveness and coverage of the base model (e.g., FPN's performance is capped by the disentanglement in base VAE latent spaces) (Wielopolski et al., 2021).
• Heavy conditioning reduces diversity by collapsing to conditional manifolds represented in unconditional generators (Chan et al., 2018).
• High-dimensional or composite conditioning may incur parameter growth or require specialized embeddings (addressed by cross-attention, LoRA, or polynomial expansions) (Chrysos et al., 2021, Li et al., 2024).

5. Advanced Topics: Multi-modal, Flexible, and Unified Conditional Generation

Recent advances address increased generality and flexibility:

• Multi-modal and cross-domain conditioning: CDCGen enables conditional synthesis in label-scarce target domains by aligning latent spaces of normalizing flows from source and target via adversarial and cycle-consistent mechanisms; attribute encoders map conditioning variables to the shared latent (Das et al., 2021).
• Unified Conditional Diffusion: UniCon achieves unified modeling of joint, conditional, inverse, and coarse conditional tasks (image-depth, edge-image, pose-image) via joint noising and parallel cross-attention with LoRA adapters; combining multiple conditioned adapters enables multi-signal fusion (Li et al., 2024).
• Continuous and high-dimensional labels: EFM establishes an inductive bias for continuous $c$7, solving a generalized PDE and using Dirichlet energy regularization for smooth interpolation and style transfer, validated on synthetic grids and latent-space MNIST (Isobe et al., 2024).
• Flexible plug-and-play: Plugin VAE architectures and flow adapters enable one-shot adaptation to new conditions without retraining the full generator, facilitating rapid and modular extension across domains (Duan et al., 2019, Wielopolski et al., 2021).

6. Applications, Metrics, and Practical Guidelines

• Applications: Controlled image synthesis (class/attribute conditioning), 3D model manipulation (rotation-consistent paired cGANs), text infilling, semantic editing, data augmentation for low-data regimes, causal decision making, and experimental design.
• Metrics:
  • Fidelity: FID, IS, PPL
  • Diversity/coverage: LPIPS-GT, Inception diversity, Wasserstein error
  • Task-specific: mIoU for segmentation, AbsRel for depth, PCK for pose, classifier accuracy for alignment, diagnostic error for CPTs.
• Design guidelines:
  • For low-dimensional $c$8, direct embedding via additive or multiplicative injection is efficient; for spatial or high-dimensional labels, use cross-attention or polynomial expansions (Chrysos et al., 2021).
  • Plug-in flows and plugin-VAEs offer computationally efficient adaptivity (Wielopolski et al., 2021, Duan et al., 2019).
  • Joint modeling (e.g. (Li et al., 2024)) enables both direct control and multi-task extension but requires careful alignment of base distribution supports.
  • Dirichlet regularization and optimal transport couplings enable smoother generalization in continuous-$c$9 settings (Isobe et al., 2024).

7. Open Directions and Theoretical Perspectives

• Inductive bias: EFM introduces exact mass-conservation and continuity in conditional flows, avoiding hand-tuned guidance schedules (Isobe et al., 2024).
• Unified frameworks: UniCon demonstrates parameter-efficient unification of conditional tasks, supporting flexible control and multi-modal conditioning (Li et al., 2024).
• Bayesian decision and uncertainty quantification: Conditioning on partial observations with probabilistic models enables robust uncertainty quantification, supporting Bayesian experimental design and information acquisition (Garn et al., 2015, Harvey et al., 2021).
• Scalability and modularity: Plugin and LoRA-style architectures support rapid extensibility and computational efficiency as the number or complexity of conditioning variables increases (Wielopolski et al., 2021, Li et al., 2024).

A key research direction is the further integration of explicit inductive-bias mechanisms (PDE-structured flows, optimal transport, cross-attention) with scalable, modular architectures for conditional generation under rich, multi-modal, and continuous conditioning variables. The development of unified evaluation protocols for conditional coverage, semantic fidelity, and task-adaptivity remains an open area.