Distributionally Robust Data Join (DJ)

• The paper proposes a framework that integrates optimal transport, distributionally robust optimization, and statistical learning to robustly join heterogeneous, biased data sources.
• Distributionally Robust Data Join (DJ) is defined by constructing ambiguity sets as intersections of optimal transport balls around empirical distributions to control bias and error.
• The approach offers strong out-of-sample guarantees and tractable convex formulations, outperforming naive pooling and barycentric aggregation methods.

Distributionally Robust Data Join (DJ) is a principled framework for robust decision making under uncertainty when data is drawn from multiple heterogeneous—and potentially biased—sources. DJ combines optimal transport (OT) theory, distributionally robust optimization (DRO), and statistical learning, constructing ambiguity sets via multiple OT neighborhoods around the empirical distributions of each source. The approach provides a significant advance over naive pooling or barycentric aggregation by accounting for statistical error and systematic bias present in each dataset, allowing robust out-of-sample guarantees, improved statistical efficiency, and tractable convex optimization formulations (Rychener et al., 2024, &&&1&&&).

1. Formal Problem Setting and Motivation

Consider an uncertain environment where a decision variable $x \in \mathcal{X}$ must be chosen to minimize an expected loss $\ell(x,\xi)$ with respect to a target distribution $P^*$, which is unknown:

$$\min_{x\in \mathcal{X}} \,\mathbb{E}_{P^*}[\ell(x, \xi)]$$

Instead of samples from $P^*$, one observes samples from $K$ distinct data sources with unknown, biased distributions $P_1, ..., P_K$. Each $P_k$ is represented by $N_k$ i.i.d. samples $\{\xi_{k,j}\}_{j=1}^{N_k}$, yielding empirical measures $$\hat{P}_k = \frac{1}{N_k} \sum_{j=1}^{N_k}\delta_{\xi_{k,j}}$$. The central challenge is to leverage all available information robustly without inflating bias (via naive pooling) or variance (by using target-like data alone) (Rychener et al., 2024).

The two-source variant, Distributionally Robust Data Join (DRDJ), considers both labeled data $(x,y)$ and unlabeled auxiliary data $(x,z)$, formulating a joint predictor that hedges against distributional uncertainty across both feature sets (Awasthi et al., 2022).

2. Construction of Ambiguity Sets via OT Intersections

The DJ approach measures the discrepancy between any candidate distribution $Q$ and each empirical source $\hat{P}_k$ using an optimal transport cost $W_c$:

$$W_c(Q, \hat{P}_k) = \inf_{\pi\in\Pi(Q, \hat{P}_k)}\int_{\Xi\times\Xi}c(\xi, \xi')d\pi(\xi, \xi')$$

Here $c(\cdot, \cdot)$ denotes a lower-semicontinuous cost (e.g., $c(\xi, \xi') = \|\xi - \xi'\|_p^p$), and $\Pi(Q,\hat{P}_k)$ denotes couplings between $Q$ and $\hat{P}_k$. The ambiguity set is defined as the intersection of $K$ OT-balls of radii $\rho_k$ centered at each $\hat{P}_k$:

$$\mathcal{U} = \bigcap_{k=1}^K \mathbb{B}_{\rho_k}(\hat{P}_k), \quad \mathbb{B}_{\rho_k}(\hat{P}_k) = \{Q\in \mathcal{P}(\Xi): W_c(Q, \hat{P}_k)\leq \rho_k\}$$

This intersection constrains any plausible $Q$ to be simultaneously close (in OT sense) to each empirical source, substantially shrinking the ambiguity set as $K$ increases (Rychener et al., 2024). In DRDJ, the ambiguity region is:

$$U(r_1, r_2) = \{Q \in \mathcal{P}(X \times Z \times Y): W_{d_{XY}}(Q_{X,Y}, \hat{P}_L) \le r_1,\ W_{d_{XZ}}(Q_{X,Z}, \hat{P}_U)\le r_2\}$$

where $d_{XY}, d_{XZ}$ are transport costs on the respective product spaces (Awasthi et al., 2022).

3. Distributionally Robust Optimization Formulation

The core DJ optimization problem seeks a decision (or predictor) robust to all distributions in $\mathcal{U}$:

$$\min_{x\in\mathcal{X}} \sup_{Q\in\mathcal{U}}\ \mathbb{E}_Q[\ell(x,\xi)]$$

For $K=1$, this reduces to standard Wasserstein DRO. For $K>1$, DJ enforces compatibility with all sources, preventing overfitting to any single biased dataset (Rychener et al., 2024).

In the DRDJ setting, the aim is:

$$\min_{f\in\mathcal{F}} \max_{Q\in U(r_1, r_2)}\ \mathbb{E}_{(X, Z, Y)\sim Q}[\ell(f(X), Y)]$$

This minimax structure generalizes classical DRO to simultaneously hedge against multiple potential distributional shifts (Awasthi et al., 2022).

4. Calibration of Ball Radii and Quantification of Bias

The radii $\rho_k$ encode prior knowledge on the magnitude of distributional shift (bias) between the source $P_k$ and the unknown $P^*$, plus sampling error:

$$\rho_k = \text{bias}_k + \text{statistical\_tol}_k,\quad \text{statistical\_tol}_k = O(N_k^{-1/2})$$

These can be set using either frequentist or Bayesian approaches. In the Bayesian case, a prior is placed on the OT distance $d_k = W_c(P^*, P_k)$, and $\rho_k$ is chosen as a quantile of the corresponding posterior, yielding explicit probabilistic coverage guarantees:

$P^*(Q\in \mathcal{U}) \ge 1 - \sum_k \beta_k$

A plausible implication is that tighter and better-informed priors over $\rho_k$ reduce conservatism while maintaining robustness (Rychener et al., 2024).

5. Statistical and Theoretical Guarantees

The DJ framework inherits strong out-of-sample guarantees from DRO literature. When balls are calibrated so that $$P_k^{N_k}[W_c(P_k, \hat{P}_k)\leq \delta_k]\geq 1- \beta_k$$, then:

$$P_{data}[P^* \in \cap_{k=1}^K \mathbb{B}_{\rho_k}(\hat{P}_k)] \ge 1 - \sum_{k=1}^K \beta_k$$

Under mild convexity or concavity of $\ell$ in $\xi$, the regret relative to $P^*$ shrinks as $K$ increases even if individual $P_k$ remain biased, provided the priors on $\rho_k$ are informative. This contrasts sharply with naive pooling or barycentric combination, which may not offer robustness if biases persist (Rychener et al., 2024, Lau et al., 2022).

In the two-source DRDJ setting, out-of-sample generalization bounds show the test risk is within $O(1/\sqrt{n})+O(r_1 + r_2)$ of the training DRO objective. Feasibility is guaranteed if $W_X(\hat{P}_{L,X}, \hat{P}_{U,X}) \leq r_1 + r_2$ (Awasthi et al., 2022).

6. Convex Reformulation and Computational Tractability

Under standard conditions ($\Xi$ convex, $c(\cdot,\cdot)$ convex, loss $\ell(x, \xi)$ piecewise-concave in $\xi$), the inner maximization over the intersection of OT-balls admits a finite convex dual formulation. Lagrange multipliers $\lambda_k\ge 0$ and dual variables $\gamma_{k, j}$ allow the suprema to be reformulated as:

$$\inf_{\lambda \geq 0, \gamma} \sum_{k=1}^K \rho_k \lambda_k + \sum_{k, j} p_{k, j} \gamma_{k, j}$$

subject to

$$\sup_{\xi\in\Xi} [\ell(x, \xi) - \sum_{k=1}^K \lambda_k c(\xi, \xi_{k, j_k}) ] \leq \sum_{k=1}^K \gamma_{k, j_k}, \quad\forall (j_1,\ldots,j_K)$$

The final program has $O(N_1 \cdots N_K)$ constraints—exponential in $K$ but polynomial in data sizes. For fixed $K$ or fixed dimension $d$, these constraints reduce to a manageable number using cell decompositions, and the program remains tractable (Rychener et al., 2024). In practice, first-order or interior-point solvers efficiently handle the convex formulations arising in both DJ and DRDJ (Awasthi et al., 2022).

7. Comparison to Wasserstein Barycenter Approaches

Wasserstein barycentric DRO (WBDRO) centers a single OT-ball at the Wasserstein barycenter of the sources, defined as the measure minimizing $\sum_k \lambda_k W_p^p(\nu, \hat{P}_k)$. The corresponding DRO problem hedges against all distributions within a ball centered at this barycenter (Lau et al., 2022).

Key differences are summarized below:

Feature	DJ/DRDJ (Rychener et al., 2024, Awasthi et al., 2022)	WBDRO (Lau et al., 2022)
Ambiguity Set	Intersection of $K$ OT-balls	Single OT-ball at barycenter
Bias Handling	Explicit, via radii $\rho_k$	Averaged by barycentric aggregation
Shrinking with $K$	Yes, ambiguity set shrinks as $K$ grows	Shrinking depends on barycenter
Robustness to Heterogeneity	Maintained for persistent bias	May not be robust if biases persist

A plausible implication is that DJ is preferable when source biases are significant and persistent, as it prevents any one source from unduly influencing the robust decision (Rychener et al., 2024).

8. Practical Considerations and Empirical Results

DJ and DRDJ require choosing radii and cost parameters, commonly via cross-validation. As domain knowledge about source shifts is incorporated, the ambiguity set can be finely tuned. Empirical results indicate DRDJ can outperform both single-anchor DRO and regularized predictors, particularly when unlabeled auxiliary data are closer to the test distribution (Awasthi et al., 2022).

Extensive tractability studies and experimental analyses confirm that DJ/DRDJ combine the statistical benefits of multi-source leverage with principled robust optimization, admitting strong theoretical guarantees and efficient convex programming formulations (Rychener et al., 2024, Awasthi et al., 2022).