---
title: Distributional Certificates
url: https://www.emergentmind.com/topics/distributional-certificates
type: topic
---

# Distributional Certificates

Searching arXiv for the cited papers and closely related work on distributional certificates.
First, locating the focal paper on distributional individual fairness.
Now searching for complementary papers on distributional certificates in robustness, fairness, and formal verification.
Distributional certificates are formal guarantees for properties evaluated not at a single input or along a single trajectory, but over a set of admissible probability distributions. In machine learning and formal verification, they typically take the form of a computable upper bound on a worst-case expected loss, violation, or specification value over an uncertainty set such as a Wasserstein ball or a Hellinger ball. A central instance is the certification of distributional individual fairness, where one seeks an upper bound on the worst-case expected local fairness violation under all distributions within a prescribed Wasserstein radius of an empirical distribution [2311.11911]. Closely related uses include robustness certificates for smoothed classifiers [2010.10987], fairness certificates under Hellinger-bounded shift [2205.15494], PAC-Bayesian certificates for posterior-averaged adversarial risk [2502.14298], sound-and-complete proof rules for distributional $\omega$-regular specifications in MDPs [2507.04286], tight Wasserstein robustness certificates for deep networks [2510.10000], and concave certificates based on least concave majorants of growth-rate functions [2601.01311].

## 1. General notion and scope

In the broader machine-learning literature, a distributional certificate is typically an upper bound on some worst-case risk or property over a set of distributions, often a Wasserstein or $f$-divergence ball. What varies across subfields is the certified functional: worst-case expected loss, worst-case misclassification probability, expected local fairness violation, posterior-averaged adversarial risk, or satisfaction of a temporal specification over a sequence of distributions [2311.11911].

| Setting | Certified object | Representative paper |
|---|---|---|
| Neural-network individual fairness | $\sup_{P^\star \in \mathcal{B}_\gamma(\hat P^0)} \mathbb{E}_{x\sim P^\star}[\mathcal{I}(f^\theta,x,\delta)]$ | [2311.11911] |
| Smoothed classifiers | Worst-case adversarial population loss over input distributions | [2010.10987] |
| Group-fair deployment shift | Worst-case loss over fair distributions within Hellinger distance $\rho$ | [2205.15494] |
| Bayesian linear regression | PAC-Bayesian bounds on $\mathbb{E}_{\theta\sim\rho}[R_\epsilon(\theta)]$ | [2502.14298] |
| Distributional MDP verification | Existence of ranking/invariant proof objects for $\omega$-regular specifications | [2507.04286] |
| Wasserstein DRO for deep nets | Certified upper bound on worst-case expected loss over $\Omega_1$ | [2510.10000] |
| General DRO geometry | Concave upper bound $\mathcal{C}_{\Delta_\theta^{\max}}(\epsilon)$ | [2601.01311] |

A recurrent misconception is to identify a distributional certificate with a global worst-case guarantee over the entire input domain. The fairness literature makes the distinction explicit: global individual fairness requires $\sup_x \mathcal{I}(f^\theta,x,\delta)\le \epsilon$, whereas distributional certification only constrains the expected violation over distributions inside a Wasserstein neighborhood. This is weaker than global certification, but substantially more scalable and more tightly tied to likely deployment shifts [2311.11911].

Another important distinction concerns what is being certified. Some papers certify a deterministic predictor under distribution shift, whereas PAC-Bayesian work certifies a randomized predictor by bounding posterior-averaged risk. In that setting, the guarantee is on $\mathbb{E}_{\theta\sim\rho}[R(\theta)]$ or $\mathbb{E}_{\theta\sim\rho}[R_\epsilon(\theta)]$, not on a single point estimate [2502.14298].

## 2. Mathematical formulations

A common starting point is a nominal distribution, usually the empirical distribution $\hat P^0=\frac{1}{n}\sum_{i=1}^n \delta_{x^{(i)}}$, and an uncertainty set around it. In the fairness and robustness literature, the uncertainty set is frequently a Wasserstein ball,
\[
\mathcal{B}_\gamma(P^0)=\{P^\star: W_p(P^0,P^\star)\le \gamma\},
\]
or, for robustness of smoothed classifiers,
\[
\{P: W_c(P\otimes Z,P_0)\le \rho\},
\]
where $Z=\mathcal N(0,\sigma^2 I)$ models Gaussian smoothing [2311.11911; 2010.10987].

For empirical measures, Wasserstein DRO often reduces to finite-dimensional perturbation variables. In distributional individual fairness, any $P^\star\in\mathcal{B}_\gamma(\hat P^0)$ can be represented by moving each atom $x^{(i)}$ to $x^{(i)}+\phi^{(i)}$ under the constraint
\[
\frac{1}{n}\sum_{i=1}^n \|\phi^{(i)}\|^p \le \gamma^p.
\]
This turns the infinite-dimensional supremum over distributions into an optimization over sample-wise perturbations [2311.11911].

The property of interest is then inserted into the expectation. For individual fairness, the local fairness map is
\[
\mathcal{I}(f^\theta,x,\delta)=\inf\{\epsilon\ge 0:\forall x''\text{ with }d_{\text{fair}}(x,x'')\le\delta,\ |f^\theta(x)-f^\theta(x'')|\le \epsilon\},
\]
and distributional individual fairness requires
\[
\sup_{P^\star\in\mathcal{B}_\gamma(P^0)}\mathbb{E}_{x\sim P^\star}[\mathcal{I}(f^\theta,x,\delta)]\le \epsilon.
\]
A distributional certificate is therefore a computable upper bound $\overline{\epsilon}$ on that left-hand side [2311.11911].

Other domains instantiate the same pattern differently. In certified fairness with subpopulation decomposition, the uncertainty set is defined by a Hellinger ball together with a fairness constraint that the shifted distribution $\mathcal Q$ is a fair base-rate distribution:
\[
\max_{\mathcal Q}\ \mathbb E_{(X,Y)\sim\mathcal Q}[\ell(h_\theta(X),Y)]
\quad\text{s.t.}\quad
H(\mathcal P,\mathcal Q)\le \rho,\ \mathcal Q\text{ is a fair distribution}.
\]
In Bayesian linear regression, the certified quantity is the posterior-averaged adversarial risk
\[
\mathbb{E}_{\theta\sim\rho}[R_\epsilon(\theta)],
\]
bounded by PAC-Bayesian terms involving empirical loss, $\mathrm{KL}(\rho\|\pi)$, and sub-gamma parameters of the adversarial loss [2205.15494; 2502.14298].

A broader geometric abstraction replaces Lipschitz constants by growth-rate functions. For loss $\bm{l}(z;\theta)$, the individual growth rate is
\[
\Delta_\theta(z,t)=\sup_{z':\,d(z',z)\le t}\{\bm{l}(z';\theta)-\bm{l}(z;\theta)\},
\]
and the central certificate is the least concave majorant of the maximal empirical rate. This yields a bound of the form
\[
\mathcal R_p(\epsilon)\le \hat{\mathcal R}+\mathcal C_{\Delta_\theta^{\max}}(\epsilon),
\]
which applies even to non-Lipschitz and non-differentiable losses [2601.01311].

## 3. Distributional individual fairness

The paper “Certification of Distributional Individual Fairness” formalizes the fairness case around a metric-based notion of individual fairness for neural networks. With input space $X\subseteq\mathbb R^m$, output space $Y\subseteq\mathbb R^k$, network $f^\theta:X\to Y$, and fair distance metric $d_{\text{fair}}$, local $\delta$–$\epsilon$ individual fairness at input $x'$ is defined by
\[
\forall x''\text{ s.t. }d_{\text{fair}}(x',x'')\le \delta
\Longrightarrow
|f^\theta(x')-f^\theta(x'')|\le \epsilon.
\]
The quantity $\mathcal I(f^\theta,x,\delta)$ is the smallest such $\epsilon$, so $\mathcal I=0$ corresponds to perfect local fairness [2311.11911].

Distributional individual fairness moves from pointwise control to distributional control under shift. Let $P^0$ be the data-generating distribution and $\mathcal B_\gamma(P^0)$ the Wasserstein ball of radius $\gamma$. Then $f^\theta$ is distributionally $\gamma$–$\delta$–$\epsilon$ individually fair if
\[
\sup_{P^\star\in\mathcal B_\gamma(P^0)}
\mathbb E_{x\sim P^\star}[\mathcal I(f^\theta,x,\delta)]
\le \epsilon.
\]
Because $P^0$ is unknown, the paper replaces it by the empirical distribution $\hat P^0$ and certifies the resulting empirical Wasserstein-ball property [2311.11911].

In this setting, a certificate is a sound upper bound
\[
\sup_{P^\star\in\mathcal B_\gamma(\hat P^0)}
\mathbb E_{x\sim P^\star}[\mathcal I(f^\theta,x,\delta)]
\le \overline{\epsilon}.
\]
The exact optimum $\epsilon^\star$ is given by
\[
\epsilon^\star=
\max_{\{\phi^{(i)}\}}
\frac{1}{n}\sum_{i=1}^n \mathcal I(f^\theta,x^{(i)}+\phi^{(i)},\delta)
\quad\text{s.t.}\quad
\frac{1}{n}\sum_{i=1}^n \|\phi^{(i)}\|^p\le \gamma^p,
\]
but this is intractable. The paper therefore computes a lower bound $\underline{\epsilon}\le \epsilon^\star$ and a certified upper bound $\overline{\epsilon}\ge \epsilon^\star$, with the latter constituting the actual distributional certificate [2311.11911].

Conceptually, this work positions distributional fairness certification between two older extremes. Pointwise IF certificates upper-bound $\mathcal I(f^\theta,x,\delta)$ at a single input, while global IF certificates attempt to prove $\sup_x \mathcal I(f^\theta,x,\delta)\le \epsilon$ over the entire domain. The former are too local for deployment guarantees; the latter, especially in MILP formulations associated with Benussi et al. and Khedr et al., do not scale beyond very small networks. Distributional certificates sacrifice full-domain worst-case coverage in exchange for guarantees over likely distribution shifts [2311.11911].

## 4. Constructive techniques

A striking feature of the literature is the variety of proof mechanisms used to make distributional certificates computable. In distributional individual fairness, the key relaxation is an orthotope over-approximation of a Mahalanobis fair ball:
\[
\{x'': d_S(x',x'')\le \delta\}\subseteq \prod_{j=1}^m [x'_j-\delta\sqrt{d_j},\,x'_j+\delta\sqrt{d_j}],
\]
where $d=\mathrm{diag}(S)$. This turns an ellipsoidal fairness region into an axis-aligned box, after which interval bound propagation yields a sound local bound
\[
\overline{\mathcal I(f^\theta,x',\delta)}=|y^U-y^L|.
\]
The distributional upper bound is then obtained by quasi-convex optimization over radius variables $\varphi^{(i)}$:
\[
\overline{\epsilon}=
\max_{\{\varphi^{(i)}\ge 0\}}
\frac{1}{n}\sum_{i=1}^n \overline{\mathcal I(f^\theta,x^{(i)},\delta+\varphi^{(i)})}
\quad\text{s.t.}\quad
\frac{1}{n}\sum_{i=1}^n |\varphi^{(i)}|^p\le \gamma^p.
\]
The same framework is also used as a training-time regularizer through F-IBP, L-DIF, and U-DIF objectives [2311.11911].

In smoothed classification robustness, the dominant construction is dual and penalized rather than box-based. The central quantity
\[
\sup_{P:W_c(P\otimes Z,P_0)\le \rho}\mathbb E_{P\otimes Z}[\ell(\theta;s)]
\]
is upper-bounded by
\[
\gamma\rho+\mathbb E_{P_0}[\Phi_\gamma(\theta;x_0)],
\qquad
\Phi_\gamma(\theta;x_0)=\sup_{x\in\mathcal X}\mathbb E_z[\ell(\theta;x+z)-\gamma c(x+z,x_0)].
\]
This certificate leads to the Noisy Adversarial Learning procedure, whose inner optimization becomes strongly concave when the transportation cost is strongly convex and the smoothed loss is $2M$-smooth [2010.10987].

In fairness under Hellinger-bounded shift, tractability comes from subpopulation decomposition. Writing the data as a mixture over subpopulations indexed by $(X_s,Y)$ yields an exact Hellinger decomposition
\[
H(\mathcal P,\mathcal Q)\le \rho
\iff
\sum_i \sqrt{p_i q_i}\,(1-H^2(\mathcal P_i,\mathcal Q_i))\ge 1-\rho^2.
\]
Under sensitive shifting, where $\mathcal Q_{s,y}=\mathcal P_{s,y}$, the certificate reduces to a tight convex program in mixture weights $k_s$ and $r_y$. Under general shifting, the paper combines this decomposition with the Weber et al. bound, a change of variables, and a grid-based convex relaxation to obtain a non-tight but tractable certificate [2205.15494].

For Wasserstein DRO in deep networks, a different line of work uses exact Lipschitz geometry. For ReLU networks, the certificate is
\[
\mathbb E_{\mathbb P_N}[\ell(Z;\theta)] + \bm{L}\epsilon,
\qquad
\bm{L}=2^{1/s}\max_{\bm D\in\mathcal D_\mathcal X}\|J_{\bm D}\|_{r\to s},
\]
with matching lower bounds based on explicit adversarial distributions and a Wasserstein Distributional Attack that constructs a candidate worst-case distribution in $\Omega_1$ [2510.10000]. The 2026 concave-certificate framework generalizes the same ambition: instead of a global Lipschitz constant, it uses the least concave majorant of the growth-rate function and introduces adversarial scores as tractable layer-wise relaxations for neural networks [2601.01311].

Outside statistical learning, distributional certificates may be symbolic proof objects rather than numerical risk bounds. For MDPs under the distribution-transformer semantics, a certificate is a pair $(C,I)$ consisting of a distributional Büchi ranking function and a distributional invariant over the product of the MDP dynamics and a nondeterministic Büchi automaton. The ranking decreases on non-accepting automaton states and stays nonnegative on accepting states, yielding sound-and-complete proof rules for distributional $\omega$-regular specifications [2507.04286].

## 5. Guarantees, tightness, and empirical behavior

The central desideratum is soundness: a certificate must never under-estimate the worst-case quantity it claims to bound. The fairness paper proves that the orthotope relaxation and IBP yield a sound upper bound on local individual fairness, that the empirical Wasserstein-ball supremum is equivalent to the perturbation optimization over $\phi^{(i)}$, and that the quasi-convex radius optimization produces $\overline{\epsilon}\ge \epsilon^\star$. It also establishes Hölder continuity for the aggregated objective under a Lipschitz or Hölder condition and gives finite-sample guarantees via Hoeffding-type inequalities, so that with $n=\Omega(\tau^{-2}\log 1/\lambda)$ the empirical certificate deviates from the population value by at most $\tau$ with probability at least $1-\lambda$ [2311.11911].

Tightness varies substantially across constructions. In the sensitive-shifting fairness setting, the subpopulation-decomposition program is both convex and tight: its optimum equals the supremum in the original certification problem. In the general-shifting setting, by contrast, the certificate is explicitly non-tight because it inherits slack from the Weber per-subpopulation bound and the grid relaxation, though it remains non-trivial in experiments on six real-world datasets [2205.15494].

For smoothed classifiers, the upper bound based on $\gamma\rho+\mathbb E_{P_0}[\Phi_\gamma]$ is accompanied by two important comparison results: adding Gaussian noise yields a worst-case distributional loss no larger than that of the base classifier, and the smoothed surrogate certificate is no worse than the standard base-classifier Wasserstein DRO certificate derived in the style of Sinha et al. [2010.10987]. In the Bayesian linear-regression setting, the guarantees are rigorous PAC-Bayesian generalization certificates for standard and adversarial risks of both the standard posterior and the adversarially robust posterior, expressed through explicit data-fit terms, KL terms, and sub-gamma tail parameters [2502.14298].

The 2025 WDRO paper sharpens the notion of tightness by proving two-sided bounds
\[
\mathbb E_{\mathbb P_N}[\ell]+\bm l\epsilon
\le
\sup_{\mathbb P:\mathcal W_{d,1}(\mathbb P,\mathbb P_N)\le \epsilon}\mathbb E_{\mathbb P}[\ell]
\le
\mathbb E_{\mathbb P_N}[\ell]+\bm L\epsilon,
\]
and then giving sufficient conditions under which $\bm l=\bm L$, so that the Lipschitz certificate is exact [2510.10000]. The 2026 concave-certificate framework makes a related claim at a more abstract level: the least concave majorant produces a tight bound on DR risk that remains applicable when the loss is non-Lipschitz and non-differentiable [2601.01311].

Empirically, the fairness literature emphasizes that soundness need not imply vacuity. Distributional IF certificates were evaluated under geographic and temporal shifts on Folktables data, where the lower bound tracked the observed worst-case empirical DIF and the upper bound was conservative but typically within a factor of order $1$ above the empirical worst-case, especially for models trained with DIF regularization [2311.11911]. The Hellinger-based fairness framework reported that sensitive-shifting certificates closely tracked the upper envelope of sampled fair distributions, while general-shifting certificates stayed clearly below trivial bounds and tightened further under non-skewness constraints [2205.15494].

## 6. Limitations and open directions

A recurring limitation is conservatism induced by tractable relaxations. In distributional individual fairness, the orthotope may be much larger than the true fair ball, especially in high dimension, and IBP is known to loosen on deep networks; as a result, $\overline{\mathcal I}$ and $\overline{\epsilon}$ can be conservative for large $\gamma$ or deep models, even though soundness is preserved [2311.11911]. In the Hellinger-based framework, the general-shifting certificate is not tight, and the number of convex subproblems grows as $T^{SC}$, which becomes costly beyond the common binary-sensitive, binary-label setting [2205.15494].

Another limitation is that certificates are usually scenario-specific. Distributional IF addresses individual fairness, not group fairness; extending distributional robustness guarantees to group-fairness metrics is identified as an open path. The Hellinger-based fairness work restricts attention to fair base-rate distributions and does not guarantee performance under arbitrary shifts. Smoothed-classifier certificates depend on choosing $\gamma$ and $\sigma$, and their practical implementation uses approximate inner maximization. PAC-Bayesian adversarial certificates are derived only for Bayesian linear regression under a Gaussian model with $\ell_2$ perturbations. The MDP verification framework currently targets distributionally memoryless strategies and affine templates. The exact-Lipschitz WDRO analysis for deep networks is limited in practice by the difficulty of exploring large numbers of reachable ReLU activation patterns. The concave-certificate framework still requires tractable upper bounds on the growth-rate function for large deep models [2010.10987; 2502.14298; 2507.04286; 2510.10000; 2601.01311].

The uncertainty model itself is also an open design choice. Wasserstein distance dominates much of the literature, but alternatives already appear. The fairness certification framework based on subpopulation decomposition uses Hellinger distance and notes implications for total variation. The distributional IF paper explicitly mentions that only Wasserstein distance is considered and that other distances, including $f$-divergences, optimal transport with different costs, or learned transport costs, could define alternative uncertainty sets. The 2026 framework is deliberately stated for general measurable costs $d$, which suggests a route toward broader transport-based certificates, though the paper does not claim a universal computational method [2205.15494; 2311.11911; 2601.01311].

Taken together, these works support a precise view of distributional certificates as a family of proof techniques for uncertainty over distributions rather than points. In one branch, they certify fairness, robustness, or posterior risk by upper-bounding a worst-case expectation over a distributional ambiguity set. In another, they certify temporal properties of systems that transform distributions over time. The unifying theme is not a single algorithm, but a common shift in semantics: robustness, fairness, and verification are framed as properties of distributional perturbations or distributional trajectories, and the certificate is the sound object that upper-bounds, or otherwise proves, the desired property under that distributional view [2311.11911; 2507.04286].

Source: https://www.emergentmind.com/topics/distributional-certificates