Conditional Instance Normalization

• Conditional Instance Normalization is a technique that uses discrete, domain-indexed affine parameters to enable many-to-many, zero-pair image translation.
• It modulates decoder activations via specific scale-and-shift vectors, reducing parameters while boosting generalization across unseen domain pairs.
• Empirical results demonstrate improved performance in tasks like depth-to-semantic translation with significantly fewer parameters compared to traditional models.

Conditional Instance Normalization (CIN)—when specialized as Domain Conditional Normalization (DCN)—is a normalization technique that enables a single encoder–decoder network to perform image-to-image translation across multiple domains. In DCN, domain-specific affine parameters modulate the decoder’s activations, facilitating zero-pair translation: mapping between two domains for which no paired training data exists, but which each have paired training data with a shared third domain. This approach leverages discrete, domain-indexed scale-and-shift vectors in normalization layers, yielding parameter efficiency and enhanced generalization to unseen domain translations (Shukla et al., 2020).

1. Formal Definition and Mechanism

Given a convolutional feature tensor $x \in \mathbb{R}^{B \times C \times H \times W}$ at any decoder layer (batch size $B$, channels $C$, height $H$, width $W$), DCN first computes per-sample, per-channel spatial statistics:

$$\mu_{i,c} = \frac{1}{HW}\sum_{h=1}^H \sum_{w=1}^W x_{i,c,h,w}$$

$$\sigma^2_{i,c} = \frac{1}{HW}\sum_{h=1}^H \sum_{w=1}^W (x_{i,c,h,w} - \mu_{i,c})^2$$

The normalized activation is:

$$\hat{x}_{i,c,h,w} = \frac{x_{i,c,h,w} - \mu_{i,c}}{\sqrt{\sigma^2_{i,c} + \epsilon}}$$

Following normalization, domain-specific scale and shift parameters are applied:

$$y^{(d)}_{i,c,h,w} = \gamma^{(d)}_c \cdot \hat{x}_{i,c,h,w} + \beta^{(d)}_c$$

where $\gamma^{(d)}, \beta^{(d)} \in \mathbb{R}^C$ are learned per-domain affine parameters for each domain index $d \in \{1,\ldots,N\}$. At each decoder layer, the parameters for the current target domain are used.

In DCN-0, there exist $N$ sets of $(\gamma, \beta)$—one per domain. In the full DCN design, the conditioning depends on both input and output domain pairs, potentially yielding $N^2$ parameter sets.

2. Relation to Conditional Instance/Batch Normalization

Conditional Instance Normalization (Adaptive IN; Huang & Belongie, 2017) typically operates by generating $(\gamma, \beta)$ via a continuous style embedding $z$ passed through an MLP. Conditional Batch Normalization (Dumoulin et al., 2017) models categorical information by associating learned $(\gamma^{(c)}, \beta^{(c)})$ tables with each class $c$.

DCN follows the categorical conditioning paradigm but assigns $(\gamma, \beta)$ to domains or domain pairs instead of style classes. Unlike continuous style codes, DCN employs solely discrete (one-hot) domain identifiers: scale-and-shift vectors are indexed by domain or input–output domain pairs, never by continuous style vectors. This design supports explicit many-to-many translation between discrete visual domains.

3. Architectures and Implementation Variants

DCN-0: Output Domain Conditional Normalization with Latent-Space Invariance

• Encoder $E$ maps any domain input to a shared latent representation $z = E(x)$.
• A domain-classifier $C$ predicts the input domain from $z$; its gradient is reversed (Ganin et al., 2016), enforcing domain invariance via an auxiliary loss $\mathcal{L}_{cls}$ weighted by $\lambda_{cls}$.
• Decoder $G$ uses only $$(\gamma^{(d_{\text{out}})}, \beta^{(d_{\text{out}})})$$ for the target domain at each normalization layer.
• During training, only the $(\gamma, \beta)$ sets for paired domains (RGB–Depth, RGB–Semantic) are updated.

DCN: Input–Output Domain Conditional Normalization

• The encoder $E$ is not enforced to be domain-invariant.
• Each batch normalization layer in $G$ is conditioned on the domain pair $(d_{\text{in}}, d_{\text{out}})$.
• For three domains (R=RGB, D=Depth, S=Semantic), there are $3 \times 3 = 9$ possible $(\gamma, \beta)$ sets.
• During training, only seen domain pairs (R→D, D→R, R→S, S→R) update parameters. Unseen pairs (D→S, S→D) are handled via a pseudo-pair loss $\mathcal{L}_{pseudo}$, which enforces output similarity between translations sharing a common target domain from different sources:

$$\mathcal{L}_{pseudo} = \mathbb{E}_{r,d}\left\| G_{D \rightarrow S}(E(d)) - G_{R \rightarrow S}(E(r)) \right\|_1$$

Only parameters for (D→S)/(S→D) are updated, with all others frozen.

Both architectures employ a consistent ResNet-based decoder; only the index and routing of normalization parameter slices differ.

4. Zero-Pair Training Protocol

The training protocol leverages available paired datasets (12k RGB–Depth, 12k RGB–Semantic; each split disjoint), but requires generalization to unseen domain pairs (1k test Depth–Semantic). The training losses employed include:

• Relativistic LSGAN adversarial loss $\mathcal{L}_{GAN}$ with direction-specific discriminators
• Reconstruction $\ell_1$ loss on paired data:

$$\mathcal{L}_{L1}^{R \rightarrow D} = \lambda_D \mathbb{E}_{r,d} \left\| G_{R \rightarrow D}(E(r)) - d \right\|_1 + \lambda_R \mathbb{E}_{r,d} \left\| G_{D \rightarrow R}(E(d)) - r \right\|_1$$

• Identity loss $\mathcal{L}_{idt}$:

$$\mathcal{L}_{idt}^{R \rightarrow D} = \lambda_D \mathbb{E}_d \left\| G_{D \rightarrow D}(E(d)) - d \right\|_1 + \lambda_R \mathbb{E}_r \left\| G_{R \rightarrow R}(E(r)) - r \right\|_1$$

• Domain-classifier loss ($\lambda_{cls}\mathcal{L}_{cls}$) for DCN-0, and pseudo-pair loss ($\mathcal{L}_{pseudo}$) with weight-freezing for DCN.

Training alternates mini-batches from both paired sets, with a learning rate of $2 \times 10^{-4}$ for the initial 120k iterations, followed by linear decay to zero by 240k iterations.

5. Empirical Evaluation and Results

DCN variants are evaluated quantitatively and qualitatively on zero-pair translation tasks, notably for depth-to-semantic segmentation. Key metrics include mean Intersection-over-Union (mIoU) and Pixel Accuracy.

Synscapes Dataset (Depth→Semantic)

Method	mIoU (%)	Pixel Acc (%)	Parameters (M)
CycleGAN	25.5	65.2	—
Cascaded 2×pix2pix	48.3	87.2	45.5
Mix-and-Match Net (M²Net)	29.1	75.3	27.1
DCN-0	40.9	84.5	11.4
DCN	54.3	89.7	11.4

Ablation: Depth→Semantic (Effect on mIoU)

• DCN-0 w/o domain classifier loss ($\mathcal{L}_{cls}$): 16.5%
• DCN w/o pseudo-pair loss: 2.1%
• DCN w/o weight freezing: 50.1% (down from 54.3%)

SceneNetRGBD Indoor (Depth→Semantic)

Method	Pixel Acc (%)	mIoU (%)
pix2pix	68.7	22.5
M²Net	39.5	7.1
DCN	59.0	10.7

Qualitatively, DCN sharpens object boundaries and recovers small classes more effectively than baseline models, achieving competitive accuracy and segmentation quality with just 11.4M parameters—nearly four times fewer than pix2pix cascades.

6. Extensions and Interpretations

Domain Conditional Normalization extends the applicability of conditional batch normalization paradigms to allow many-to-many, zero-pair domain translation by (i) associating $(\gamma, \beta)$ parameters with domain indices or domain-pair indices, and (ii) employing domain-invariant latent representations (DCN-0) or pseudo-pair loss (DCN) for learning unseen translation channels. A plausible implication is that such explicit discrete conditioning and modular training of normalization parameters support rapid generalization to novel domain pairs without excess model complexity or interference with previously learned translations.

7. Significance and Applications

DCN enables a single, compact encoder–decoder framework to generalize in image-to-image translation across modalities such as RGB, depth, and semantic segmentation without requiring direct paired data for each domain pair. This substantially reduces the parameter count relative to cascaded or ensemble approaches and improves both qualitative and quantitative outcomes in zero-pair and many-to-many translation settings. The technique is broadly applicable in contexts where sample efficiency, cross-domain generalization, and model compactness are required (Shukla et al., 2020).