Conditionally Invariant Components (CICs)

• Conditionally Invariant Components (CICs) are latent features that stay constant across domains when conditioned on true labels, ensuring robust predictive performance.
• Estimation methods such as conditional invariant penalties and importance-weighted reweighting tackle covariate and label shifts by aligning class-conditional distributions.
• Empirical results show CIC-based models boost classification accuracy and mitigate bias in synthetic datasets, image benchmarks, and high-dimensional biological data.

Conditionally Invariant Components (CICs) are latent features or computations that remain invariant across domains when conditioned on ground-truth labels. In statistical learning, specifically domain adaptation (DA) and representation learning, CICs encapsulate the axiom that certain predictive signals are preserved when considering label information, even if domain (environment, batch, or other nuisance interventions) statistics otherwise differ. Their identification, estimation, and exploitation have emerged as fundamental to robust generalization across disparate data sources, particularly under combined covariate and label shifts or when domain-specific nuisances confound learning. CICs are rigorously formulated and operationalized in both domain adaptation theory (Wu et al., 2023) and causal representation learning for high-dimensional biological systems (Aliee et al., 2023).

1. Formal Definitions and Conditional Invariance

A function $\phi:\mathbb{R}^p \rightarrow \mathbb{R}^q$ is a conditionally invariant feature mapping if for $M$ labeled source domains—each with distribution $P^{(m)}_{X,Y}$—and a target domain $P_{X,Y}$, the conditional distribution of transformed features given the label remains invariant for all domains and all label classes:

$$P^{(m)}_{\phi(X)\mid Y=y} = P_{\phi(X)\mid Y=y} \quad \forall m, \forall y.$$

In the case $q=1$, $\phi(X)$ is termed a conditionally invariant component (CIC). Under such mappings, classifiers $h = g \circ \phi$ can achieve minimax target risk among all functions enforcing this conditional invariance constraint (Wu et al., 2023).

In the latent variable setting of biological data, the latent space $z$ is partitioned into $z_I$ ("invariant") and $z_S$ ("spurious"), representing, respectively, meaningful signal and nuisance factors. The prior factorizes as $$p(z_I, z_S \mid d, e) = p(z_I \mid d) p(z_S \mid e)$$, where $d$ encodes domain knowledge (e.g., disease state) and $e$ captures environmental or nuisance variation (e.g., batch or lab). Crucially, under the conditional independence $y \perp e \mid z_I$, the target $y$ becomes invariant to unwanted variation given $z_I$ (Aliee et al., 2023).

2. Estimation Methodologies

Domain Adaptation: Conditional Invariant Penalty (CIP)

CICs are estimated by penalizing divergences in the class-conditional distributions of $\phi(X)$ across all pairs of source domains:

$$\min_{g \in \mathcal{G},\, \phi \in \Phi} \frac{1}{M} \sum_{m=1}^M \widehat{R}^{(m)}(g \circ \phi) + \lambda_{\rm CIP} \frac{1}{L M^2}\sum_{y=1}^L \sum_{m \neq m'} D\left(\widehat{P}^{(m)}_{\phi(X) | Y = y},\, \widehat{P}^{(m')}_{\phi(X) | Y = y}\right),$$

where $D$ is typically the Maximum Mean Discrepancy (MMD) and the empirical source risk is minimized alongside an alignment term (Wu et al., 2023).

Biological Data: Conditional Priors and Total Correlation

In the variational autoencoder (VAE) architecture, invariant ($z_I$) and spurious ($z_S$) latent variables are enforced with factorized priors, an explicit total correlation penalty $$KL(q_\phi(z_I, z_S|u) \parallel q_\phi(z_I|u)q_\phi(z_S|u))$$ to decouple invariant and spurious structure, and domain-conditioned or environment-conditioned exponential family priors (Aliee et al., 2023). The learning objective integrates the ELBO, score-matching for non-normalized priors, and total correlation penalties.

3. CICs Under Covariate and Label Shift

Traditional DA assumptions such as covariate shift ($P^{(m)}(X) \neq P(X)$, $P^{(m)}(Y \mid X) = P(Y \mid X)$) or label shift ($P^{(m)}(Y) \neq P(Y)$) are generalized through CICs. Importance-weighted CIP (IW-CIP) addresses label-shift scenarios by reweighting source examples:

$w^{(m)}_j = \frac{P(Y = j)}{P^{(m)}(Y = j)},$

propagating these weights in both the classification loss and the alignment penalty. IW-CIP proceeds by:

• Solving the unweighted CIP to obtain initial representations.
• Estimating class-conditional confusion matrices on sources and target.
• Solving a weighted CIP using estimated class-importance weights, yielding refined invariance and improved risk guarantees even when both covariate and label shift occur jointly (Wu et al., 2023).

4. Theoretical Guarantees and Generalization Bounds

Under the sole assumption of the existence of CICs, the target risk for any classifier $h = g \circ \phi$ can be upper-bounded by:

• The importance-weighted average source risk,
• The estimation error in class-importance weights,
• The worst-case class-conditional mismatch term $\Psi_G(\phi)$.

Theorems ensure that the IW-CIP estimator achieves target risk within additive factors involving the conditional invariance penalty, weight estimation errors, and finite-sample complexity terms. Specifically:

$$R(\widehat{h}) \leq R(h^\star) + \lambda_{\rm CIP} + 2 \max_m \|w^{(m)} - \widehat{w}^{(m)}\|_\infty + \Psi_G(\widehat{\phi}) + \mathcal{O}(1/\sqrt{n_m})$$

for the finite-sample estimator ($\widehat{h}$) and oracle CIC-classifier ($h^\star$) (Wu et al., 2023). In high-dimensional linear models, risk convergence for CIC-classifiers is exponentially fast in the difference between ambient and CIC dimension.

5. Roles of CICs in Diagnosing and Improving Domain Adaptation

CICs have diagnostic and corrective roles in DA algorithms:

• Risk Detection: Using a CIC-based reference classifier ($h_{\rm ref}$) with low source error enables provable upper bounds on the difference between target and source risks for any classifier $h$ based solely on observable discrepancies between $h$ and $h_{\rm ref}$, in both source and target domains. This allows for data-dependent failure certificates when DA algorithms such as DIP fail due to non-invariant features (Wu et al., 2023).
• Failure Correction: Standard domain-invariant projection (DIP) can align marginal representations but admit "label-flipping" features, where class labels are permuted in the target domain. Incorporating known CICs into JointDIP, which matches the joint distribution of a learned feature and a CIC across domains, eliminates this pathology and guarantees target risk bounded above by that of the best available CIC-classifier.

6. Applications and Empirical Results

CIC-based methodologies have been validated across synthetic and real-world datasets:

• Synthetic linear structural equation models: IW-CIP corrects label shifts, and JointDIP prevents label-flipping; risk detection identifies DIP failure scenarios (Wu et al., 2023).
• Rotated MNIST, CelebA, and Camelyon17: JointDIP- and IW-CIP-based models outperform or correct DIP, achieving higher target accuracy (e.g., 93.5% for JointDIP-Pool with MMD on Rotated MNIST, compared to ~90% for DIP). On Camelyon17, JointDIP-Pool increases AUC from ~81% (DIP-Pool) to ~82.7%, and up to ~91.9% on a held-out target set.
• Single-cell genomics: InVAE-bio, employing a partitioned latent space for CICs, achieves superior biological conservation and batch correction as measured by external benchmarks such as scIB, and enables accurate classification of cell types on held-out donors (accuracy 0.653 vs 0.579 for scANVI on hematopoiesis). In lung cancer datasets, the approach removes batch effects without erasing intrinsic biological structure (Aliee et al., 2023).

7. Domain-Specific Frameworks and Outlook

In biological data integration, conditionally invariant representation learning operationalizes CICs by:

• Utilizing hierarchical conditional priors (e.g., non-factorized for invariant, factorized for nuisance latents).
• Enforcing independence through total correlation penalties,
• Exploiting causal assumptions (e.g., $y \perp e \mid z_I$) to guarantee invariance of predictions to domain-specific variation.

Such approaches are empirically validated with robust metrics, including both classification and unsupervised measures (ASW, kBET, NMI, ARI, F1), and visual inspection of representations. A plausible implication is that CICs offer a unifying principle for cross-domain generalization and bias removal, supported by theoretical risk bounds as well as domain-specific interpretability and performance (Aliee et al., 2023, Wu et al., 2023).