DAGBag Procedure: Ensemble DAG Learning

• DAGBag procedure is a robust method that learns the structure of directed acyclic graphs by aggregating candidate graphs from bootstrap resampled data.
• It minimizes Structural Hamming Distance to select only consistently recurring edges, thereby reducing false positives and overfitting.
• The procedure employs efficient hill climbing and regularization techniques suitable for high-dimensional applications such as genomics and network analysis.

The DAGBag procedure is a statistical framework for learning the structure of directed acyclic graphs (DAGs) from data, using bootstrap aggregating (bagging) principles to aggregate multiple candidate graph structures into a robust estimate. Its core aim is to reduce the variance and overfitting typical of high-dimensional DAG learning procedures, particularly the excessive detection of false positive edges caused by data noise and small sample sizes. Bagging allows DAGBag to deliver higher stability and accuracy in discovering the conditional independence (or potential causality) relationships among variables, with practical computational techniques that scale to large problems.

1. Ensemble Construction via Bootstrap Aggregation

The DAGBag procedure generates an ensemble of candidate DAGs by repeatedly resampling the original dataset with replacement (bootstrap resampling). For each bootstrap sample, a base DAG learning method—typically a score-based approach such as hill climbing with the Bayesian Information Criterion (BIC)—is applied to learn a candidate graph. The resulting set

$$\mathbb{G}^e = \{\mathcal{G}_1, \mathcal{G}_2, \dots, \mathcal{G}_B\}$$

captures the variability of learned graph structures due to data fluctuations. This ensemble effectively represents the distribution of DAGs that could be plausibly learned from the observed data under resampling, separating recurring features (edges) from those induced by sample noise.

This approach is especially critical in high-dimension, low-sample settings ($p \gg n$), where overfitting causes conventional DAG learners to propose excessive edges. Bootstrapping exposes which edge relations persist across resamples, providing raw input for subsequent aggregation to stabilize the final estimate.

2. Aggregation via Structural Hamming Distance Metrics

Once the ensemble is created, the DAGBag procedure aggregates these candidate graphs into a single representative DAG by minimizing a suitable distance metric. The key principle is to find a "median graph," i.e., a DAG that is, on average, closest to all graphs in $\mathbb{G}^e$ under a chosen structural metric.

The principal family of metrics used is the Structural Hamming Distance (SHD), which measures the minimum number of edge deletions, additions, or reversals required to transform one DAG into another. For adjacency matrices $A$ and $\tilde{A}$,

$$d_{\text{SHD}}(\mathcal{G}, \tilde{\mathcal{G}}) = \sum_{i,j} |A(i,j) - \tilde{A}(i,j)|$$

The DAGBag procedure generalizes SHD to allow variable penalties for edge reversal, yielding

$$d_{\text{GSHD}(\alpha)} \text{ where } \alpha > 0 \text{ is penalty for reversal}$$

The aggregation target is

$$\mathcal{G}^* = \arg\min_{\mathcal{G} \in \mathbb{G}(\mathbb{V})} \frac{1}{B} \sum_{b=1}^B d(\mathcal{G}, \mathcal{G}_b)$$

Crucially, when using SHD (with edge reversal counted as two operations), the aggregation score is available in a decomposable additive form: $$\text{score.SHD}(\mathcal{G}; \mathbb{G}^e) = \sum_{e \in \mathbb{E}(\mathcal{G})} (1-2p_e) + C$$ where $p_e$ is the empirical frequency with which edge $e$ appears in the ensemble, and $C$ does not depend on $\mathcal{G}$. This super‐decomposability means only edges with selection frequency $p_e > 0.5$ contribute to lowering the score; thus, aggregation reduces to greedily adding edges in decreasing frequency order (subject to acyclicity).

3. Efficient Hill Climbing Search and Regularization

DAGBag implements an optimized hill climbing algorithm to efficiently search the super-exponential space of DAGs, targeting the SHD-based aggregation score.

Key aspects of the algorithm include:

• Local decomposability: Local changes (add, delete, reverse edges) only affect the score of nodes immediately involved; this permits rapid incremental score updating.
• Acyclicity checks: After each operation, the algorithm only checks for cycles when parent/child sets change, minimizing redundant computation.
• Edge pre-screening: Since only edges with $p_e > 0.5$ can improve the score, the search is restricted to high-frequency candidates.
• Early stopping and random restarts: The search halts when no operation decreases the score by more than a threshold $\epsilon$; although random restarts are supported, they often provide no benefit due to the stabilizing effect of aggregation.

For problems with up to 1000 nodes and sample size 250, the procedure performs thousands of steps in minutes—several orders of magnitude faster than naïve approaches—making it computationally competitive in high-dimensional regimes.

4. Variance Reduction and Overfitting Control in High Dimensions

High-dimensional DAG structure learning suffers from an accumulation of spurious (noise-driven) edges when using naive estimators, especially when $p \gtrsim n$. DAGBag overcomes this by enforcing stability via both ensemble frequency and built-in complexity penalties:

• Bootstrap averaging: Noisy edges appear inconsistently across bootstraps (yielding low $p_e$) and are thus excluded by the aggregation rules (selection only if $p_e > 0.5$).
• Regularized aggregation score: Explicit penalization of edges with $p_e \leq 0.5$ acts analogously to BIC, shrinking the solution towards sparsity and true positives. Empirical results in practical applications confirm that DAGBag greatly reduces the number of false positive edges while retaining stable, repeated dependencies.

5. Application Domains and Use Cases

The DAGBag framework is suitable for learning probabilistic graphical models in diverse high-dimensional scientific and social science contexts where structure discovery is central. Representative domains include:

• Genomic and gene regulatory networks: Inferring robust connections among hundreds or thousands of genes, prioritizing control of false discoveries.
• Causal structure learning in biomedical research: Determining directional relations among biomarkers, imaging data, and clinical measurements.
• Social or financial networks: Extracting dependencies and influence patterns in systems with many interacting agents but limited observations.

Whenever standard DAG learning yields overly dense, unstable graphs, DAGBag can act as a variance reduction and model regularization procedure, improving interpretability and scientific validity.

6. R Package Implementation and Practical Considerations

The DAGBag methodology is operationalized in the "dagbag" R package. The package supports:

• Pluggable base methods (e.g., BIC-based hill climbing).
• Bootstrap ensemble construction and aggregation as described.
• Computation of selection frequencies for all directed edges.
• Efficient hill climbing with super-decomposable aggregation scores.
• Option to select alternative SHD variants via the reversal penalty parameter $\alpha$.

This implementation incorporates all score updating and acyclicity strategies described, enabling practical use on problems with hundreds or thousands of variables. As a result, it matches or exceeds the speed of other state-of-the-art structure learning routines for large-scale applications.

7. Theoretical and Methodological Significance

DAGBag advances the state of probabilistic graphical structure learning by systematically addressing the instability and overfitting issues endemic to high-dimensional settings. By formalizing graph aggregation under SHD-type metrics and integrating regularization at the consensus graph level, it achieves variance reduction analogous to classifier bagging; crucially, the use of super-decomposable score functions makes the solution computationally tractable.

In summary, the DAGBag procedure enables robust, scalable DAG structure learning through ensemble averaging and aggregation scoring, underpinned by efficient optimization and rigorous penalization schemes, with demonstrated utility in genomics, network science, and causal inference contexts (Wang et al., 2014).