Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation

Abstract: Reliable generalization in conditional latent variable models requires understanding both identifiability and extrapolation: how observed variation across attributes determines latent structure, and how that structure determines distributions at unseen attributes. However, existing identifiability and extrapolation guarantees are largely model-specific, with separate analyses in nonlinear ICA, causal representation learning, perturbation modeling, and related conditional latent variable models. We introduce concept modulation models (CMMs), an attribute-indexed class of conditional generative models with structure $A\to Λ\to C\to X$, where attributes select modulators, modulators induce latent concept laws, and concepts generate observed features. CMMs lift transition-based identifiability to conditional settings by showing that feature agreement on observed attributes induces a latent concept transition constrained by the CMM class. We express these constraints through attribute potentials, log-density ratios between attribute-conditioned concept laws, separating the generic lifting step from model-specific rigidity arguments. The same potentials control extrapolation: agreement at unseen attributes holds exactly when the transported attribute-potential identities extend to those attributes. This yields algebraic extrapolation criteria, identifies the common potential-based proof objects behind several existing identifiability and extrapolation results, and, when combined with the model-specific rigidity arguments in those works, recovers their stated conclusions.

• The paper introduces a general conditional generative framework (CMM) that unifies identifiability and extrapolation in weakly supervised representation learning.
• It systematically constrains latent concept structures using observed attribute modulation through indexing, concept, and mixing kernels.
• The approach recovers insights from nonlinear ICA and causal representation learning while providing novel extrapolation criteria.

Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation

Overview

The paper "Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation" (2606.18509) introduces a general conditional generative modeling framework—Concept Modulation Models (CMMs)—which systematically addresses identifiability and extrapolation in weakly supervised representation learning. The CMM framework unifies and abstracts analysis from domains such as nonlinear ICA, causal representation learning, and perturbation modeling by encapsulating how observed attribute-driven variations constrain latent structure and generalization to new conditions.

Concept Modulation Model Structure

CMMs are formulated as conditional generative models with the graphical structure $A \to \modrv \to C \to X$, where:

• $A$ is an observed attribute (environment, intervention, conditioning context, etc.).
• $\modrv$ is a modulator, specifying how the attribute affects latent concept distributions.
• $C$ is the latent concept variable.
• $X$ is the observed feature or data.

Formally, a model consists of:

• An indexing kernel $\bQ$ mapping $A$ to a modulator $\modrv$.
• A concept modulation kernel $\bB$ mapping the modulator to a concept distribution.
• A mixing kernel $\bK$ mapping concepts to observed features.

The key modeling assumption is that $A$0 is shared across a class of models, while $A$1 and $A$2 capture freedom in how attributes index modulators and how concepts generate observations.

Transition-Based Identifiability in Conditional Models

The central technical innovation is the extension of the transition-based framework for latent identifiability (Squires et al., 27 Apr 2026) to conditional settings. The main question is: given observed $A$3 for $A$4, what is the set of possible latent transitions $A$5 that could relate models indistinguishable on these conditions? The authors show that, for CMMs, feature equivalence on $A$6 admits only those latent transitions that satisfy two compatibility constraints:

• Mixing-side validity: Existence of a kernel $A$7 such that $A$8.
• Concept-side validity: For $A$9, existence of $\modrv$0 such that $\modrv$1 for all $\modrv$2.

These combine in a transition-intersection criterion: identifiability up to a group $\modrv$3 is guaranteed if the intersection of all valid transitions lies within $\modrv$4.

Attribute Potentials: Algebraic Proof Objects for Identifiability and Extrapolation

A primary contribution is the identification of attribute potentials as universal objects mediating identifiability and extrapolation. The attribute potential for attributes $\modrv$5 is defined as:

$\modrv$6

These log-density ratios isolate the contrastive effect of varying $\modrv$7 on the concept law.

The authors prove that valid latent transitions $\modrv$8 must preserve attribute potentials across observed attributes:

$\modrv$9

This directly connects to model-specific identifiability proofs (e.g., in nonlinear ICA, exponential family latent variable models, CRL) which all, in effect, leverage contrast preservation in some form. The constraint extends to extrapolation: agreement on transported attribute potentials at new attributes is equivalent to distributional agreement at those attributes.

Extrapolation Theorems

The framework yields novel, general extrapolation criteria. The critical result is that, once a latent transition $C$0 is fixed by feature equivalence on $C$1, extrapolation to $C$2 holds if and only if the attribute-potential identities induced by $C$3 extend to all $C$4:

$C$5

For structured attribute spaces, such as those arising in causal or combinatorial intervention regimes, the paper gives algebraic criteria under which extrapolation is guaranteed—e.g., affine dependence of potentials or combinatorial interaction designs.

Recovery and Unification of Prior Results

The CMM framework generalizes and recovers identifiability and extrapolation for:

• Nonlinear ICA/iVAE: Component-wise contrast in conditionally independent latent sources recovers the standard permutation-and-componentwise ambiguity class.
• Causal Representation Learning: Environmental or interventional contrasts and local mechanism shifts produce the identificational and extrapolational results of recent CRL literature, including nonparametric and score-based variants.
• Perturbation Models: Conditional mean-shifted Gaussian concept models recover orthogonal invariance in effect matrices and extrapolation to unseen perturbations [von2025representation].

The paper also demonstrates partial identifiability when contrast subspace conditions are violated, characterizing exactly which components of latent structure remain determined by the data.

Implications and Future Directions

Theoretically, this work reframes identifiability in weakly supervised settings as a problem of algebraic potential preservation under transitions, separating the proof objects (attribute potentials) from the specific geometry of individual model classes. It reveals algebraic, model-independent reasons behind the success of many contrast-based identifiability guarantees, placing extrapolation on the same technical footing.

Practically, the CMM framework highlights the importance of controlled attribute modulation in representation learning for generalization. Designing attribute structures (environments, interventions, conditions) to maximize the rank or information content in observed potentials directly relates to the range of extrapolation and uniqueness of learned representations.

For future developments, this approach suggests:

• Extending CMMs to non-overlapping or singular-support scenarios (e.g., deterministic interventions) and to settings with overlapping support in concept laws.
• Leveraging attribute potential criteria as a tool for evaluating or designing architectures in neurosymbolic and constraint-based learning, where weak indirect supervision is common.
• Analyzing sample complexity, stability, and algorithmic aspects (e.g., optimization under potential constraints) in the context of practical, data-driven learning systems.

Conclusion

The CMM framework introduced in this paper (2606.18509) constitutes a unifying, algebraic theory for studying identifiability and extrapolation in conditional latent variable models. By making explicit the role of attribute-indexed potential functions and latent transitions, it both generalizes and sharpens our understanding of when and how weak supervision induces uniquely determined and extrapolatable latent representations in machine learning.