Mixture of DAGs Framework

• Mixture of DAGs is a framework that represents observed distributions as mixtures over several latent directed acyclic graphs, accommodating heterogeneity and non-stationarity.
• It extends classical graphical models using tools like mixture d-separation, union MAGs, and auxiliary nodes to capture context-dependent causal mechanisms.
• The approach underpins practical algorithms such as CADIM and EM-based structure search for causal discovery in biomedical, genomic, and spatial applications.

A mixture of Directed Acyclic Graphs (DAGs) is a probabilistic modeling framework in which the observed distribution is generated as a mixture over several latent causal structures, each encoded as a DAG on the same set of variables. Unlike models assuming a single, static DAG, a mixture of DAGs accommodates heterogeneity, non-stationarity, and context-dependent causal mechanisms. This enables modeling of multi-modal, time-evolving, or population-stratified processes where distinct regimes are governed by different acyclic graphs and the observed data marginally blends these regimes. The mixture of DAGs paradigm underpins modern approaches to identifiability theory, causal discovery algorithms, and spatial random field modeling, and is foundational for understanding the limitations and capabilities of learning causal structure in complex systems.

1. Formal Model and Graphical Representations

Let $V = \{1, \ldots, n\}$ be the node set. A mixture of DAGs specifies $K \geq 2$ component DAGs $G_\ell = (V, E_\ell)$, $\ell = 1, \ldots, K$. Each component induces a Markov factorization

$$p_\ell(x_1, \ldots, x_n) = \prod_{i \in V} p_\ell(x_i | x_{\text{pa}_\ell(i)}),$$

with parent sets $\text{pa}_\ell(i) \subset V$. A latent variable $L \in \{1, \ldots, K\}$ samples the component according to weights $r(\ell) = \mathbb{P}(L=\ell)$, yielding the observed mixture

$p_m(x) = \sum_{\ell=1}^K r(\ell) \, p_\ell(x).$

To characterize (conditional) independencies in $p_m(x)$, one defines the mixture DAG $G_{m, I}$ for each post-intervention I (see Section 3 below), and associated summary graphs such as the union MAG and mother graphs for abstracting the union of CI-relations (e.g., (Strobl, 2019, Saeed et al., 2020)).

Key derived sets:

• Mixture-parent set: $(i) := \bigcup_{\ell=1}^K \text{pa}_\ell(i)$.
• True edges: $E_t = \{\,j \to i : j \in (i)\,\}$ (present in at least one component).
• Inseparable pairs ($E_i$), emergent pairs ($E_e$), and mechanism-change set $\Delta$ track lack of separability and structural/parametric inconsistencies across components (Varıcı et al., 2024).

2. Conditional Independence, Markov Properties, and Identifiability

The conditional independence structure induced by a mixture of DAGs does not correspond, in general, to any single DAG. Several theoretical devices have been developed:

• Mixture d-separation: For any collection of DAGs $\{G^{(\ell)}\}$, the mother graph $\mathcal{M}$ places them side by side, extending classical d-separation to "mixture d-separation." A path is an $m$-path if it exists in any component (possibly using mixture colliders). The global Markov property states $A$ and $B$ are $m$-d-separated given $C$ in $\mathcal{M}$ if and only if $X_A \perp X_B | X_C$ in the mixture (Strobl, 2019).
• Mixture DAG Construction: Introduction of auxiliary nodes (e.g., a root $y$ pointing into all nodes with mechanism variation per (Varıcı et al., 2024)) allows the use of d-separation to test identifiability after interventions and to model how intervention effects propagate in the mixture structure.
• Union MAG: Merging the individual maximal ancestral graphs (MAGs) arising from each component via their Markov equivalence class, with directed or bidirected edges indicating presence and orientation of causal and confounding components (Saeed et al., 2020).
• Local Detectability: The existence of true edges (i.e., whether $j \to i \in E_t$) is testable via an intervention set $I$ such that post-intervention CI tests can separate inseparable pairs—subject to nontrivial combinatorial bounds due to emergent dependencies.

Identifiability of edge existence under mixtures is strictly more challenging than in standard DAGs. Fundamental lower and upper bounds on minimal intervention size are established, with tightness determined by the mixture-parent set and inter-component cycles (Varıcı et al., 2024).

3. Interventional Causal Discovery: Theory and CADIM Algorithm

Discovering true edges $E_t$ in a mixture of DAGs requires specialized intervention strategies. The core theoretical results (Varıcı et al., 2024) include:

• Basic Identifiability: For inseparable pairs $(j,i)$ under $p_m$, an intervention $I$ including $j$ and blocking all $\Delta$-through paths from $j$ to $i$ in every component suffices for identifiability (Lemma 1).
• Intervention Size Bounds:
  • For arbitrary mixtures, $|(i)| + 1$ interventions suffice and are sometimes necessary.
  • For $K$-component directed tree mixtures, intervention size $K+1$ is tight.

The CADIM algorithm (Causal Discovery Algorithm for Mixtures) adaptively recovers all true edges using $O(n^2)$ interventions, outlined as:

• Step 1 (Ancestor identification): For each node $i$, perform a single-node intervention; estimate $(i)$.
• Step 2 (Cycle breaking): For each $i$, find the minimal set $B(i)$ intersecting all ancestor cycles (cyclic complexity $\tau_i$); for each ancestor $j$ perform an intervention on $B(i) \cup \{j\}$.
• Step 3 (Layering): Cluster ancestors topologically given breakages.
• Step 4 (True parent identification): For each topological layer, test $j$ by intervention on previously found parents $\cup B(i) \cup \{j\}$.

CADIM achieves optimal intervention size whenever the subgraph of ancestors is acyclic. When cycles exist, the increase in intervention size is bounded by the cyclic complexity number $\tau_i$ (the minimal cycle-breaking set size among ancestors of $i$).

4. Algorithmic and Statistical Learning of Mixtures

Learning mixture-of-DAGs models from data involves both parameter and structure recovery. The canonical approach (Thiesson et al., 2013) is:

• Expectation-Maximization (EM): Treat the latent component variable as missing data, iterating between computing soft assignments (E-step) and parameter updates (M-step).
• Structure Search: Interleaved with EM, searches the space of DAG structures for each component, typically via local score optimization and component-specific sufficient statistics. The "Cheeseman-Stutz" large-sample marginal likelihood approximation is used for model selection and to efficiently score candidate structures.
• Model Selection: The number of components $K$ is chosen by maximizing marginal likelihood or via cross-validated prediction.

This strategy is computationally feasible via decomposition of complete-data scores and restricting parent-set sizes.

Statistical models such as MDGM (Mixture of Directed Graphical Models) exploit this paradigm to obtain scalable approximate posteriors for spatial random fields over graphs, leveraging collections of compatible DAGs to preserve template adjacencies and computational tractability (Carter et al., 2024).

5. Markov Properties, Summary and Union Graphs

The union structure induced by a mixture of DAGs has distinct graphical properties:

• The summary graph $S$ (or union MAG $M_\cup$) overlays all component DAGs, encoding all adjacencies as undirected or partially directed edges. Edges that reverse direction across components are rendered undirected; edges consistently oriented across all components retain their directionality (Strobl, 2019).
• Conditional independence is determined by mixture d-separation in the summary graph, which is computationally efficient and scalable compared to the full mixture/mother graph.
• In the spatial modeling context, unions over strongly or weakly compatible DAG classes (e.g., all rooted spanning trees, all acyclic orientations) can recover the full adjacency structure of a template undirected graph (Carter et al., 2024).

These properties are critical for constraint-based discovery algorithms, as empirical CI tests on pooled data correspond to mixture d-separations in the summary graph.

6. Applications, Empirical Validation, and Examples

Mixture of DAGs frameworks are applied in domains such as biomedicine, genomics, and spatial statistics, addressing nonstationarity, feedback, and heterogeneous populations:

• Biomedical Causal Systems: CIM (Causal Inference over Mixtures) consistently outperforms classical PC, FCI, and RFCI algorithms for edge and orientation recovery in longitudinal datasets such as the Framingham Heart Study and STAR*D, as measured by sensitivity and fallout metrics (Strobl, 2019).
• Genomics and Multi-population Analysis: FCI applied to pooled data from gene expression studies reliably identifies nodes with varying mechanisms and clusters samples according to underlying mixture component (Saeed et al., 2020).
• Discrete Spatial Random Fields: MDGM priors, when applied to spatial areal data (ecometrics), are empirically shown to recover joint posteriors that closely approximate those of true Markov random fields, with computational efficiency and correct recovery of structural features such as neighborhood clustering (Carter et al., 2024).
• Algorithmic Discovery: CADIM achieves near-perfect precision and recall in synthetic linear-Gaussian mixtures given moderate sample sizes and low cyclic complexity, confirming the tightness of theoretical bounds (Varıcı et al., 2024).

Table: Empirical Performance Metrics for Edge Orientation (Framingham Heart Study, (Strobl, 2019))

Algorithm	Sensitivity	Fallout	Distance
CIM	0.72±0.03	0.12±0.01	0.29±0.02
PC	0.70±0.04	0.24±0.02	0.41±0.03
RFCI	0.35±0.02	0.07±0.01	0.66±0.02
FCI	0.38±0.02	0.05±0.01	0.62±0.03
CCI	0.42±0.03	0.06±0.01	0.59±0.03

7. Open Problems and Extensions

Mixture of DAGs frameworks raise several fundamental questions:

• Determining minimal sufficient sets and optimal intervention designs amid persistent ambiguities and mixture-induced cycles.
• Extending mixture modeling to general classes (e.g., infinite/continuous mixtures, context-specific structure) and quantifying identifiability gaps in model recovery.
• Algorithmic improvements to learning mixtures with high cyclic complexity, large $K$, or limited interventions.
• Theoretical characterization of the faithfulness, completeness, and minimality of summary or union graphs in high-dimensional and high-heterogeneity regimes.

Ongoing research explores intervention-efficient algorithms, sharper computational lower bounds, and integration of mixture modeling with temporal, spatial, or hierarchical data structures (Varıcı et al., 2024, Strobl, 2019, Saeed et al., 2020, Carter et al., 2024).