Concomitant DAG Learning: On the Roles of Noise Adaptivity, Sparsity, and Non-negativity

Abstract: Directed acyclic graphs (DAGs) constitute a central modeling tool to enable principled reasoning about cause-effect interactions in complex systems. However, since the causal structure underlying a group of variables is often unknown and interventions may be infeasible or ethically challenging to implement, there is a need to address the task of inferring DAGs from observational data. However, most classical structure identification approaches face two key obstacles: the combinatorial challenge of enforcing acyclicity, which severely limits scalability, and identifiability challenges arising from latent confounding or heterogeneous noise. This tutorial offers an overview of recent signal processing and optimization advances that address these issues by recasting DAG structure learning as a continuous, score-based estimation problem over adjacency matrices. We begin with a didactic introduction to structural equation models and the formulation of causal graph recovery, followed by a historical survey of score-based methods ranging from early combinatorial search schemes and greedy heuristics to modern continuous frameworks that leverage smooth characterizations of acyclicity. Building on this foundation, we describe concomitant DAG estimation methods that jointly infer sparse causal structure and exogenous noise levels, improving robustness under heteroscedasticity and distribution shifts by rendering the estimator noise adaptive. All in all, the tutorial introduces readers to challenges and opportunities for signal processing research at the crossroads of causal inference, high-dimensional statistics, and scalable graph learning, while outlining emerging directions including online, nonlinear, and neural causal discovery.

• The paper introduces a concomitant noise-adaptive score function that jointly estimates DAG structure and noise levels, enhancing robustness to heteroscedasticity.
• It employs continuous optimization and explicit non-negativity constraints to simplify acyclicity enforcement and boost computational efficiency.
• Empirical results demonstrate lower SHD and SID on both synthetic and real datasets, highlighting reliable performance in noisy, high-dimensional settings.

Concomitant DAG Learning: Noise Adaptivity, Sparsity, and Non-negativity

Introduction: Causal Structure Discovery and its Obstacles

Directed acyclic graphs (DAGs) are foundational in encoding causal relationships between variables, especially in the context of Bayesian networks, structural equation models (SEMs), and their applications across biology, healthcare, genetics, and finance. The critical computational and statistical bottleneck in DAG learning from observational data lies in combinatorial constraints—particularly acyclicity—and the inherent identifiability issues, especially under limited sample sizes or Markov equivalence. While interventions can resolve identifiability, practical, ethical, or physical constraints often preclude them, making passive observational DAG learning central but NP-hard.

Recent developments exploit continuous optimization frameworks, with differentiable acyclicity constraints guiding gradient-based solvers. However, traditional score functions, commonly least-squares (LS) and lasso-like objectives, suffer from bias in the presence of heteroscedastic or unknown noise. Further, the critical regularization parameters tie directly to noise variance, complicating their practical tuning and decreasing robustness to distribution shifts. This work systematically addresses these limitations by introducing concomitant noise-adaptive score functions, jointly estimating the DAG structure and exogenous noise. Additionally, it analyzes the impact of explicit non-negativity constraints on edge weights—a realistic assumption for various domains—highlighting their favorable optimization and statistical consequences.

Background: SEMs, Identifiability, and Problem Formulation

In the linear SEM framework, each observed variable $x_i$ is generated as a linear combination of its parental variables plus exogenous noise, whose variances can be heterogeneous and whose distributions need not be Gaussian. The central problem is, given $n$ i.i.d. samples from such a model, to recover the underlying DAG by minimizing a composite score function—composed of data fidelity, regularization (e.g., sparsity), and the acyclicity constraint—over the (superexponential) space of DAGs.

Identifiability results underscore the impossibility of unique DAG recovery from observational data without additional restrictions. For instance, linear Gaussian SEMs are generally unidentifiable except under homoscedasticity or with non-Gaussian errors (LiNGAMs), while even nonlinear SEMs need further structure for unique solutions. Constraint-based techniques (e.g., the PC algorithm) target Markov equivalence classes, but rely heavily on faithfulness and robust conditional independence testing, which become statistically challenging especially with high-dimensional or limited data.

Continuous Optimization and Acyclicity: Advances and Challenges

A critical innovation has been the continuous relaxation of the acyclicity constraint, exemplified by functions such as NOTEARS' exponential trace criterion and the log-determinant-based characterization in DAGMA. These permit scalable, differentiable optimization over adjacency matrices. However, existing objective functions, dominated by OLS or lasso-regularized LS, assume known or uniform noise variance, embedding tuning parameters that require a priori knowledge or problematic cross-validation.

Concomitant Noise-Adaptivity: The CoLiDE Framework

The core methodological advance lies in generalizing the score function to simultaneously estimate DAG structure and noise levels, using concomitant estimators. For the homoscedastic case (CoLiDE-EV), the proposed loss incorporates the noise level $\sigma$ directly:

$$\min_{\mathbf{W},\sigma \geq \sigma_0} \frac{1}{2n\sigma}\|\mathbf{X} - \mathbf{W}^\top\mathbf{X}\|_F^2 + \frac{d\sigma}{2} + \lambda \|\mathbf{W}\|_1 \quad \text{s.t. } h(\mathbf{W})=0$$

Here, $h(\mathbf{W})$ enforces acyclicity (using, e.g., the DAGMA log-determinant characterization). Crucially, $\lambda$ now scales independently of the noise level, increasing practical robustness to noise-model mismatch and distribution shifts.

For the general heteroscedastic setting (CoLiDE-NV), a diagonal noise covariance is estimated jointly with the structure, yielding analogous computational complexity but increased flexibility.

The optimization scheme alternates, akin to block coordinate descent, between minimization over $\mathbf{W}$ (via first-order updates) and closed-form updates for $\sigma$ or its vector analog. This preserves full decomposability, enabling mini-batch and online estimation schemes—key for large-scale or streaming settings.

Empirical Results: Robustness across Heteroscedasticity and Structure

Empirical evaluation spans synthetic DAGs and real datasets. In high-dimensional ER4 and SF4 structures with varied node count and noise regimes, CoLiDE-NV consistently achieves lower structural Hamming distance (SHD) and intervention distance (SID) compared to state-of-the-art baselines, especially under noise heterogeneity. Notably, CoLiDE-EV remains competitive even when the homoscedastic assumption is broken, indicating algorithmic robustness and suitable inductive bias for practical applications.

Figure 2: Mean DAG recovery performance, plus/minus one standard deviation, under heteroscedastic noise for both ER4 (top row) and SF4 (bottom row) graphs with varying numbers of nodes. Each column corresponds to a different noise distribution.

CoLiDE also excels under limited data or high-noise conditions, with substantially reduced variability, as seen in the strong performance for real-world datasets such as the Sachs cell-signaling network. Tracking experiments validate the effectiveness of online and mini-batch variants, where CoLiDE's estimates closely match the full-data solution with lower computational cost.

Figure 4: Tracking performance of mini-batch stochastic gradient descent relative to the full-data CoLiDE-EV algorithm.

Exploiting Non-negativity: The NOMAD Algorithm

Non-negativity of edge weights (justified in, e.g., excitatory neural, contagion, or certain biological networks) enables a simplified acyclicity constraint, improving both the optimization landscape and theoretical properties. The log-determinant penalty can now be applied directly to the adjacency matrix (without the Hadamard product), yielding gradients that do not vanish at DAG solutions and removing KKT degeneracies. This permits the use of the method of multipliers for effective constrained minimization (NOMAD), providing more interpretable and computationally efficient solutions. NOMAD empirically achieves perfect recovery as sample size grows and demonstrates statistically benign optimization landscapes, underscoring the benefit of structural prior knowledge.

Figure 1: Recovery of ER4 DAGs with non-negative weights; shows sample complexity, median SHD, and F1-score across variations in sample size, nodes, and model parameters.

Implications, Extensions, and Future Directions

The demonstrable gains from noise-adaptive concomitant score functions and the strong empirical performance of non-negativity regularization hold broad implications for causal inference and ML. In particular, these methods bolster robustness to distribution shift, enable reliable operation in heteroscedastic regimes, and provide computational scalability for high-dimensional graphs. The ability to seamlessly integrate first-order solvers and online (or mini-batch) updates opens the avenue for streaming causal discovery, critical for applications in monitoring, finance, and adaptive systems.

Theoretical challenges remain, given the inescapable non-identifiability in linear Gaussian SEMs under observational data, yet the structure-equivalence results extend to these frameworks. Exciting directions include extending concomitant and non-negativity-constrained estimators to nonlinear models (e.g., neural network–parameterized SEMs), soft and hard interventional settings, and incorporating downstream utility (joint structured causal and predictive learning). Furthermore, cross-pollination with diffusion-based graph priors and plug-and-play optimization (e.g., DAG-PnP) may further strengthen generalization and sample efficiency.

Conclusion

This work provides rigorous methodological and empirical advances in DAG structure learning by integrating noise adaptivity, sparsity, and non-negativity constraints. Leveraging concomitant estimation confers robustness and practicality in score-based frameworks, overcoming critical limitations of conventional OLS-based approaches. Exploiting structural priors such as non-negativity yields tangible statistical and computational benefits. These developments set a foundation for reliable and scalable causal discovery under complex, noisy, and heterogeneous regimes, with substantial implications for both theory and practice in next-generation signal processing and machine learning.