Statistical Guarantees for Distributionally Robust Optimization with Optimal Transport and OT-Regularized Divergences

Abstract: We study finite-sample statistical performance guarantees for distributionally robust optimization (DRO) with optimal transport (OT) and OT-regularized divergence model neighborhoods. Specifically, we derive concentration inequalities for supervised learning via DRO-based adversarial training, as commonly employed to enhance the adversarial robustness of machine learning models. Our results apply to a wide range of OT cost functions, beyond the $p$-Wasserstein case studied by previous authors. In particular, our results are the first to: 1) cover soft-constraint norm-ball OT cost functions; soft-constraint costs have been shown empirically to enhance robustness when used in adversarial training, 2) apply to the combination of adversarial sample generation and adversarial reweighting that is induced by using OT-regularized $f$-divergence model neighborhoods; the added reweighting mechanism has also been shown empirically to further improve performance. In addition, even in the $p$-Wasserstein case, our bounds exhibit better behavior as a function of the DRO neighborhood size than previous results when applied to the adversarial setting.

• The paper derives finite-sample concentration inequalities for OT-DRO settings that quantify the excess risk between empirical and population objectives.
• It extends the DRO framework to incorporate OT-regularized f-divergences, ensuring convergence rates that match the population optimum even under adversarial training.
• The results justify advanced adversarial defenses by using covering number analyses and entropy integrals to control complexity in soft-constraint norm relaxations.

Statistical Guarantees for Distributionally Robust Optimization with OT and OT-Regularized Divergences

Introduction

This paper rigorously establishes finite-sample statistical guarantees for supervised learning models trained under Distributionally Robust Optimization (DRO) with ambiguity sets determined by Optimal Transport (OT) costs and OT-regularized $f$-divergences. The authors depart from prior literature constrained almost exclusively to $p$-Wasserstein metrics, addressing a much richer class of OT costs, including soft-constraint norm-ball relaxations important for empirical adversarial robustness. Their theoretical machinery further accommodates the new regime of OT-regularized divergence neighborhoods, where adversarial sample generation is coupled with reweighting—extending the analytical scope to hybrid loss neighborhoods empirically shown to outperform classical adversarial training.

Problem Setting and Motivation

The DRO framework regularizes population risk minimization,

$\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$

by substituting the expectation with a worst-case risk taken over a statistical neighborhood $\mathcal{U}(P)$, specified by a metric or divergence. In adversarially robust learning, the ambiguity set must account for both perturbation-based attacks (captured via OT metrics) and reweighting attacks (induced by $f$-divergences). The minimax formulation,

$$\inf_\theta\, \sup_{Q\in \mathcal{U}(P)} \mathbb{E}_Q[\mathcal{L}_\theta],$$

is computationally realized by convex dual formulations.

Empirically, adversarial robustness is enhanced by incorporating both adversarial sample generation (OT transport) and adversarial reweighting ($f$-divergences). Notably, soft-constraint relaxations of $\ell_p$ norm-balls, as advocated by [bui_UDR_2022unified], and OT-regularized $f$-divergences [birrell2025optimal], have shown practical superiority, but their statistical convergence behavior remained unquantified.

Main Results

The authors present comprehensive high-probability concentration inequalities and covering number-based generalization bounds for several novel DRO regimes relevant to adversarial learning:

1. Statistical Guarantees for General OT-DRO

They derive concentration inequalities for the excess risk between population and empirical versions of the OT-DRO problem, expressed as

$$\mathbb{P}^n\left(\Big|\inf_{\theta} \sup_{Q:C(P,Q)\leq r} \mathbb{E}_Q[\mathcal{L}_\theta] - \inf_{\theta} \sup_{Q:C(\hat P_n,Q)\leq r} \mathbb{E}_Q[\mathcal{L}_\theta] \Big| \geq \eta(n,\mathcal{G})\right) \leq \delta,$$

where the deviation $p$0 is explicit in the covering numbers of the function class and, crucially, does not blow up as the OT-ambiguity radius $p$1 or for more general OT costs (including soft-constraints). This improves upon previous generalization bounds (e.g., [lee2018minimax, azizian2023exact]) which deteriorate rapidly for fixed adversarial radii, a key regime for adversarial defense.

The authors’ treatment robustly covers OT costs of the form

$p$2

with $p$3 controlling the penalty for support shifts outside a norm ball, thus capturing the practical adversarial threat model.

2. OT-Regularized f-Divergence DRO

The analysis is extended to OT-regularized $p$4-divergence neighborhoods, where the ambiguity set is formulated as

$p$5

By duality, the empirical and population versions are shown to converge, and the excess risk is controlled with concentration rates depending on the Rademacher complexity of the corresponding composite function class. This is the first finite-sample analysis for DRO neighborhoods that combine both transport-based support attack and distributional reweighting.

The authors give comprehensive Lipschitz and covering number analyses to justify the entropy integrals arising in the concentration and generalization bounds, covering both KL and $p$6-divergence regularizations.

3. Empirical Risk Minimizer Consistency

ERM solutions to the empirical OT-DRO and OT-regularized $p$7-divergence problems are shown, with high probability, to achieve excess robust risk at the same rates as the population optimum, up to optimization error. The analysis accommodates realistic non-convex or inexact optimization as encountered in adversarially trained neural nets.

Technical Insights

• The covering number machinery explicitly quantifies how the complexity penalty depends on the shape of soft-constraint penalties $p$8 governing the OT cost, showing that for common choices ($p$9 or $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$0), the additional complexity is negligible or moderate.
• The dual analysis for the OT-regularized $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$1-divergence DRO leverages new results (including a sharp restriction of the dual variable domain) that guarantee existence and computability of the minimax solution even as the regularization parameter vanishes, circumventing technical obstacles that arise with general $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$2-divergences.
• The convergence results hold uniformly in OT neighborhood size $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$3, a critical property for real adversarial robustness applications where $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$4 is chosen by security constraints rather than by sample size.

Numerical and Analytical Highlights

• For standard parametric classes (e.g., linear, shallow nets), the entropy integral terms for covering numbers converge with the classical $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$5 rate, even with OT-regularization and without additional penalty for the robust constraints.
• Bounds generalize prior Wasserstein-1, Wasserstein-2, and KL-DRO results, removing the restrictive dependence on specific choices of cost and divergence functions.

Implications and Future Directions

The theoretical findings rigorously justify recent practical advances in robust adversarial training, including soft-constraint OT cost design and the simultaneous use of transport and divergence neighborhoods. The statistical consistency and fast convergence established here remove a key barrier to the adoption of these robustification techniques in high-stakes ML settings.

From a theoretical perspective, the duality and entropy analysis developed could spur investigation into additional composite and hybrid ambiguity sets in DRO, such as combinations with GAN-based divergences or neural-OT metrics. The primary open problem highlighted is the extension of OT-regularized $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$6-divergence statistical guarantees to regression (or, more generally, non-discrete label spaces), which presently require new analytical techniques.

Conclusion

This work provides the first generalization and finite-sample statistical bounds for DRO with highly expressive OT and OT-regularized $\inf_\theta\, \mathbb{E}_P[\mathcal{L}_\theta],$7-divergence ambiguity sets, operationalizing and justifying state-of-the-art adversarial robustness techniques in supervised learning. The results bridge the gap between post-hoc empirical validation and rigorous performance guarantees, delivering mathematical tools of immediate use in robust ML research and applications.

Reference:

"Statistical Guarantees for Distributionally Robust Optimization with Optimal Transport and OT-Regularized Divergences" (2603.27871)