---
title: Bayesian Double Machine Learning
url: https://www.emergentmind.com/topics/bayesian-double-machine-learning-bdml
type: topic
---

# Bayesian Double Machine Learning

Bayesian Double Machine Learning (BDML) is a fully Bayesian approach for causal inference in partially linear models with high-dimensional controls. It addresses the problem of regularization-induced confounding (RIC), which often plagues off-the-shelf machine learning estimators in high-dimensional settings, and modifies the double machine learning (DML) paradigm to deliver finite-sample valid inference. BDML operates by estimating a generative model that links the treatment and outcome through a reduced-form multivariate regression structure, recovering the causal effect via a ratio of covariance components, and leveraging Bayesian posterior analysis throughout [2508.12688].

## 1. Model Structure and Identifying Framework

The observed data consist of $n$ i.i.d. draws $\{(Y_i, D_i, X_i)\}_{i=1}^n$ governed by the partially linear structure:
\[
Y_i = \alpha D_i + g(X_i) + \varepsilon_i,\quad \mathbb{E}[\varepsilon_i\mid D_i, X_i] = 0,
\]
accompanied by
\[
D_i = m(X_i) + V_i,\quad \mathbb{E}[V_i\mid X_i]=0.
\]
In the linear special case, $g(X) = X^\top\beta$ and $m(X) = X^\top\gamma$, so
\[
Y_i = \alpha D_i + X_i^\top\beta + \varepsilon_i,\quad D_i = X_i^\top\gamma + V_i
\]
with $\operatorname{Cov}(\varepsilon_i, V_i) = 0$ and $X_i \in \mathbb{R}^p$, with $p$ large relative to $n$. This framework enables causal identification under the "selection-on-observables" assumption, provided an adequate control for $X_i$ is available.

## 2. Bayesian Generative Specification

BDML recasts the system into a reduced-form Seemingly Unrelated Regressions (SUR) model:
\[
\begin{pmatrix}
Y_i \\ D_i
\end{pmatrix}
= X_i
\underbrace{
\begin{pmatrix}
\delta' \\
\gamma'
\end{pmatrix}_B
}
+ 
\begin{pmatrix}
U_i \\ V_i
\end{pmatrix}
,\qquad (U_i, V_i)^\top\ \vert\ X_i \sim N_2(0, \Sigma),
\]
where $\delta = \alpha \gamma + \beta$. 
A conditionally conjugate prior is imposed:
\[
\Sigma \sim \operatorname{Inv\mbox{-}Wishart}(\nu_0, \Sigma_0),\quad
\delta \sim N_p\left(0, \frac{1}{\tau_\delta}I_p\right),\quad
\gamma \sim N_p\left(0, \frac{1}{\tau_\gamma}I_p\right).
\]
The joint likelihood for the data is
\[
L(B, \Sigma) = \prod_{i=1}^n \frac{1}{2\pi|\Sigma|^{1/2}}
\exp\left\{-\frac{1}{2}(W_i - X_i B)^\top \Sigma^{-1} (W_i - X_i B)\right\},
\]
where $W_i = (Y_i, D_i)$.

## 3. Identification via Reduced-Form Covariance

The causal parameter $\alpha$ is isolated through the relationship
\[
U_i = \varepsilon_i + \alpha V_i, \quad V_i = D_i - X_i^\top \gamma,
\]
implying the covariance structure
\[
\Sigma = \operatorname{Var}\begin{pmatrix} U_i \\ V_i \end{pmatrix}
= 
\begin{pmatrix}
\sigma_\varepsilon^2 + \alpha^2 \sigma_V^2 & \alpha \sigma_V^2 \\
\alpha \sigma_V^2 & \sigma_V^2
\end{pmatrix}.
\]
The causal effect is then
\[
\alpha = \frac{\operatorname{Cov}(U_i, V_i)}{\operatorname{Var}(V_i)} = \frac{\Sigma_{12}}{\Sigma_{22}}.
\]
Posterior inference of $\alpha$ is accomplished by sampling $\Sigma$ and mapping each posterior draw to the ratio $\alpha^{(s)} = \Sigma_{12}^{(s)} / \Sigma_{22}^{(s)}$.

## 4. Regularization-Induced Confounding and BDML’s Advantage

A naïve Bayesian regression of $Y$ on $(D, X)$ with independent priors on $\beta$, but lacking any link to $\gamma$, produces a ridge-type estimator with bias:
\[
\operatorname{Bias}(\widehat{\alpha}_\lambda)
= \widehat{\rho}^\top \bigl[I_p - (R+\lambda I_p)^{-1}R\bigr]\beta,
\]
where $\widehat{\rho} = (D^\top D)^{-1}D^\top X$ and $R$ is the $X$-residual gram matrix. This RIC may be substantial unless $\lambda$ is adaptively tuned to $D$-$X$ correlation. By contrast, the BDML reduced-form likelihood does not factor into separate regressions; independent priors on $(\delta, \gamma)$ no longer encode improper restrictions on $\beta$ and $\gamma$. The induced marginal prior on $\alpha = \Sigma_{12} / \Sigma_{22}$ is heavy-tailed and places negligible mass on zero as $p \to \infty$, thus BDML avoids RIC and its associated finite-sample bias.

## 5. Posterior Computation and Algorithmic Implementation

Closed-form expressions are available for the SUR model’s posterior distribution:
\[
B\mid\Sigma, X, W \sim N(B_n, V_n), \quad
\Sigma\mid B, X, W \sim \operatorname{Inv\mbox{-}Wishart}(\nu_0+n, \Sigma_0 + (W-XB)'(W-XB))
\]
with
\[
V_n = \left(\Sigma^{-1} \otimes X^\top X + V_0^{-1}\right)^{-1}, \quad
B_n = V_n (\Sigma^{-1} \otimes X^\top X)\,\mathrm{vec}(\hat B),
\]
$\hat B = (X^\top X)^{-1} X^\top W$. Since the marginal posterior for $\alpha$ lacks a closed form, one samples $\Sigma$ and computes $\alpha^{(s)}$ as above.

Algorithmic steps for posterior inference:
- Specify priors $\pi(\delta)$, $\pi(\gamma)$, $\pi(\Sigma)$ as detailed.
- Write the SUR likelihood for $(Y, D)\mid X$.
- Run an MCMC sampler (e.g., Stan's NUTS) to generate $\{(\delta^{(s)}, \gamma^{(s)}, \Sigma^{(s)})\}_{s=1}^S$.
- Compute $\alpha^{(s)} = \Sigma_{12}^{(s)} / \Sigma_{22}^{(s)}$. Posterior mean and credible intervals follow directly.

## 6. Theoretical Properties

Under mild regularity (Gaussian or sub-Gaussian errors, eigenvalues of $\operatorname{Var}(X_i)$ bounded, $p/n \to 0$, priors $\tau_\delta,\tau_\gamma = O(p)$), BDML satisfies:
- Consistency and $\sqrt{n}$–consistency of the posterior mean of $\alpha$ for $p = o(n^{3/4})$.
- Shrinkage bias of order $p^2/n^2$, strictly less than the naïve estimator’s $p/n$ bias.
- A Bernstein–von Mises theorem: the posterior of $\alpha$ converges in total variation to
\[
N\left(\alpha^*, \frac{1}{n} \frac{\sigma_\varepsilon^{2*}}{\sigma_V^{2*}}\right).
\]
Consequently, BDML achieves semiparametric efficiency, attaining the semiparametric information bound, and the credible intervals are (asymptotically) valid frequentist confidence intervals [2508.12688].

## 7. Simulation Evidence and Comparative Performance

In simulation studies, DiTraglia & Liu compare seven approaches under a data generating process with $X_i \sim N(0, I_p)$, $(\varepsilon_i, V_i) \sim N(0, \operatorname{diag}(\sigma_\varepsilon^2,1))$, $n=200$, $p=100$, $\alpha=2$, $\gamma = \mathbf{1}_p/\sqrt p$, and varied $\sigma_\varepsilon \in \{1,2,4\}$:
- **BDML-Basic:** conjugate SUR prior with $\operatorname{Normal}(0, 25)$ on $(\delta, \gamma)$
- **BDML-Hier:** hierarchical (heavy-tailed) prior, $\delta_j \sim t_4$, $\gamma_j \sim t_4$
- **Linero, HCPH, Naïve:** two-step Bayesian methods
- **FDML-Full, FDML-Split:** frequentist DML with ridge regression

Performance metrics are RMSE of $\alpha$, 95% interval coverage, and average confidence interval width. For $\sigma_\varepsilon=1$ (similar trends hold for larger $\sigma_\varepsilon$):

| Method             | RMSE  | 95% Coverage | CI Width |
|--------------------|-------|--------------|----------|
| BDML-Hier          | 0.09  | 94%          | 0.36     |
| BDML-Basic         | 0.11  | 93%          | 0.41     |
| Linero             | 0.10  | 93%          | 0.38     |
| Alternatives       | large bias/under-coverage/wide intervals |

BDML-Hier yields the lowest RMSE, near-nominal coverage, and the narrowest intervals, indicating superior all-around performance.

## 8. Assumptions and Limitations

Key assumptions and limitations underpinning BDML include:
- Sampling: i.i.d. draws of $(X_i, W_i)$.
- Dimensionality: $p < n$ and $p = o(n)$, specifically $p = o(n^{3/4})$ for root-$n$ consistency, $p = o(\sqrt n)$ for the Bernstein–von Mises result.
- Covariate structure: $\operatorname{Var}(X_i)$ with bounded spectrum; sub-Gaussian tails are sufficient.
- Error distribution: Gaussian or sub-Gaussian, for theoretical tractability; some mis-specification robustness.
- Prior hyperparameters scaled as $\tau_\delta, \tau_\gamma = O(p)$, ensuring vanishing shrinkage as $n \to \infty$.
- Absence of cross-fitting: BDML requires no sample splitting, unlike frequentist DML estimators; uncertainty is fully marginalized in the Bayesian framework.

BDML provides a generative likelihood-based framework for high-dimensional causal inference, balancing theoretical guarantees, practical implementation, and robust finite-sample properties [2508.12688].

Source: https://www.emergentmind.com/topics/bayesian-double-machine-learning-bdml