ERM with f-Divergence Regularization

Updated 7 August 2025

The paper's main contribution is deriving a unique closed-form solution for ERM-fDR by integrating an f-divergence penalty with the empirical risk objective.
The methodology unifies multiple divergence schemes, enabling a dual optimization framework that simplifies the infinite-dimensional problem to a one-dimensional scalar equation.
Key implications include enhanced robustness, privacy, and fairness, with applications extending to semi-supervised learning and robust risk minimization.

Empirical Risk Minimization with f-Divergence Regularization (ERM-fDR) is a broad, mathematically rigorous framework in which a conventional empirical risk objective is augmented by a penalty termed an f-divergence between a candidate predictor-induced probability measure and a reference measure. The approach unifies a wide class of regularization and robustness schemes—including those based on Kullback–Leibler divergence, total variation, χ² divergence, Jensen–Shannon divergence, among others—each prescribing distinct inductive biases or robustness properties, and has substantial applications in privacy, fairness, robustness, and semi-supervised learning.

1. Mathematical Formulation and Solution Characterization

In ERM-fDR, the optimization problem is

$\min_{P \ll Q}~ \int L_z(\theta) dP(\theta) + \lambda D_f(P\|Q)$

where $L_z(\theta)$ is the empirical risk (e.g. loss on the dataset $z$ at model parameter $\theta$ ), $Q$ is a reference probability measure over model space, $\lambda > 0$ is the regularization strength, and $D_f(P\|Q)$ denotes the f-divergence: $D_f(P\|Q) = \int f\left( \frac{dP}{dQ} \right) dQ$ with $f$ a strictly convex, differentiable function, $f(1)=0$ .

Unique Solution and Closed-form Densities

Under the mild condition that $L_z(\theta)$ 0 is strictly convex and differentiable, the unique solution $L_z(\theta)$ 1 is characterized by a density with respect to $L_z(\theta)$ 2 of the form

$L_z(\theta)$ 3

where $L_z(\theta)$ 4 denotes the inverse of the derivative of $L_z(\theta)$ 5, and the normalization term $L_z(\theta)$ 6 is chosen such that $L_z(\theta)$ 7 integrates to one: $L_z(\theta)$ 8

Specific choices for $L_z(\theta)$ 9 recover well-known schemes:

For $z$ 0, the solution (Gibbs measure) is $z$ 1;
For $z$ 2, the solution is $z$ 3;
For other $z$ 4 (e.g. Jensen–Shannon, Hellinger), the solution is dictated by the corresponding $z$ 5 (Daunas et al., 2024).

Support and Inductive Bias

A defining property—irrespective of $z$ 6—is that $z$ 7 remains strictly positive on $z$ 8, so the learned measure $z$ 9 cannot assign probability outside the support of the reference. This "support preservation" induces a hard inductive bias determined by $\theta$ 0, even dominating strong evidence from the data (Daunas et al., 2024, Daunas et al., 2023).

2. Dual Optimization and the Role of the Normalization Function

The primal ERM-fDR problem is computationally challenging due to the presence of a normalization constant. By duality, the problem can be reduced to a scalar optimization over a Lagrange multiplier $\theta$ 1, leveraging the Legendre–Fenchel (LF) transform: $\theta$ 2 The dual objective function is

$\theta$ 3

The minimizer $\theta$ 4 (the normalization function) equates to the unique $\theta$ 5 where

$\theta$ 6

This dual approach is both theoretically rigorous and highly efficient computationally, as it reduces the often infinite-dimensional problem to a one-dimensional nonlinear equation (Daunas et al., 20 Feb 2025, Daunas et al., 5 Aug 2025).

Nonlinear ODE for the Normalization Function

Through the implicit function theorem, a nonlinear ODE for the normalization constant is derived: $\theta$ 7 where $\theta$ 8 is the empirical risk under a reweighted measure $\theta$ 9, whose density with respect to $Q$ 0 is a function of $Q$ 1 and $Q$ 2. Existence and strict monotonicity are guaranteed under mild regularity assumptions, and efficient numerical schemes (e.g., root-finding, ODE solvers) can be applied (Daunas et al., 5 Aug 2025).

3. Equivalence Transformations and Relation to Constrained Problems

An important theoretical result is that for any two choices of $Q$ 3 and $Q$ 4 (strictly convex, differentiable), a transformation $Q$ 5 exists such that

$Q$ 6

with $Q$ 7 defined by

$Q$ 8

This establishes that the choice of divergence primarily induces a transformation (reweighting) on the loss function, not a fundamentally distinct regularization effect (Daunas et al., 2024).

Furthermore, under nontriviality of the loss, the solution of the regularized ERM-fDR is equivalent to that of the constrained optimization: $Q$ 9 by setting the regularization parameter $\lambda > 0$ 0 such that $\lambda > 0$ 1 (Daunas et al., 20 Feb 2025).

4. Applications: Privacy, Fairness, Robustness, and Beyond

ERM-fDR encompasses several advanced applications across machine learning:

Differential Privacy

In differential privacy, adding an $\lambda > 0$ 2-divergence term to the ERM objective enables the practitioner to control the proximity of the learned distribution to an invariant or prior, which can be leveraged to "smooth" sensitivity and improve privacy-utility tradeoffs. When extending objective perturbation mechanisms, the addition of an $\lambda > 0$ 3-divergence term requires verifying convexity, differentiability, and bounded derivatives to ensure privacy guarantees and tractable sensitivity analysis (0912.0071).

Fairness via f-divergence

Fair empirical risk minimization with $\lambda > 0$ 4-divergence regularization (notably, in the f-FERM and related frameworks) penalizes statistical dependence between predictions and sensitive features via a divergence between the joint output-sensitive distribution and the product of marginals. The variational (convex dual) form of the divergence makes unbiased stochastic optimization possible and allows for a principled treatment of fairness–accuracy tradeoffs that are robust to mini-batch size, enforceable even with nontrivial fairness constraints, and provably generalize at the $\lambda > 0$ 5 rate observed in unconstrained ERM (Baharlouei et al., 2023, Fukuchi et al., 2015).

Robustness and Tail Sensitivity

Replacing the expectation in ERM by an $\lambda > 0$ 6-divergence–induced risk measure enables "tailoring to the tails"—i.e., upweighting extreme losses—by choosing $\lambda > 0$ 7 corresponding to a risk aversion profile encoded via a reference distribution, which is tightly connected to Orlicz and Lorentz norms. Ambiguity sets defined by $\lambda > 0$ 8-divergence lead to distributionally robust optimization schemes; for example, KL-divergence ambiguity sets result in solutions with subexponential tail control (Fröhlich et al., 2022, Coppens et al., 2023).

Semi-supervised and Noisy Label Learning

In semi-supervised self-training, divergence-based empirical risk functions—using $\lambda > 0$ 9-divergence or α-Rényi divergence—yield losses that are notably robust to noisy pseudo-labels (due to boundedness or insensitivity of certain divergences) and can regularize soft assignments toward uniformity, improving performance and stability over classical methods (Aminian et al., 2024).

Overparameterized Regimes and Functional Risk

Functional Risk Minimization (FRM) interprets ERM-fDR as a special case where loss is defined over distributions in function space. The natural f-divergence regularizer in FRM penalizes complexity in the space of functions, providing an explicit mechanism for choosing "simplest" solutions among many overfit predictors (Alet et al., 2024).

5. Technical and Practical Considerations

Several critical technical ingredients arise in ERM-fDR:

Support Restriction and Inductive Bias

Regardless of the empirical data, the solution support is always contained within the reference measure's support. This can be highly desirable (if $D_f(P\|Q)$ 0 encodes valid prior knowledge), but may result in severe bias if $D_f(P\|Q)$ 1 is poorly specified (Daunas et al., 2023, Daunas et al., 2024).

Choice of Divergence

The selection of $D_f(P\|Q)$ 2 influences both mathematical tractability and practical properties. For instance, KL-regularization yields exponential weights (Gibbs measure), while total variation or bounded divergences may confer robustness to noise (as in fairness- and robust-risk contexts) (Aminian et al., 2024).

Duality and Algorithmic Solvability

Utilizing the dual formulation not only yields theoretical insight but also greatly simplifies optimization and normalization via a one-dimensional root-finding (on $D_f(P\|Q)$ 3), and the ODE-based approach provides an efficient algorithm under broad regularity (Daunas et al., 5 Aug 2025, Daunas et al., 20 Feb 2025).

Table 1: Primal and Dual Formulations in ERM-fDR

Formulation	Optimization Variable	Constraint/Normalization
Primal	Probability measure $D_f(P\\|Q)$ 4	$D_f(P\\|Q)$ 5; support in $D_f(P\\|Q)$ 6
Dual	Scalar $D_f(P\\|Q)$ 7	$D_f(P\\|Q)$ 8

Conditions for Existence and Uniqueness

Strict convexity and differentiability of $D_f(P\|Q)$ 9, together with "separability" of $D_f(P\|Q) = \int f\left( \frac{dP}{dQ} \right) dQ$ 0 (nonconstancy over $D_f(P\|Q) = \int f\left( \frac{dP}{dQ} \right) dQ$ 1's support) ensure solution existence, uniqueness, normalization function monotonicity, and dual-primal equivalence (Daunas et al., 20 Feb 2025, Daunas et al., 5 Aug 2025).

6. Relationships to Alternative Regularization and Extensions

ERM-fDR subsumes and generalizes classical regularization paradigms:

L₂-regularization can be seen as a specific (quadratic) f-divergence;
Reverse relative entropy and Jeffrey’s divergence produce differing bias and tail properties through their corresponding solutions (Daunas et al., 2024);
Alternative divergences induce equivalent regularization upon suitable transformation of the empirical risk (Daunas et al., 2024).

Type-II regularization, reversing arguments of the KL-divergence, leads to equivalent solutions (relative to Type-I) after a log transformation of the empirical risk. However, the bias toward the prior remains intrinsic in both cases (Daunas et al., 2023).

7. Summary and Theoretical Implications

ERM-fDR provides a formally unified and flexible foundation for incorporating structured inductive bias, robustness, privacy, and fairness in risk minimization problems. Its key theoretical implications include

uniqueness and explicit closed-form density of solutions (in terms of $D_f(P\|Q) = \int f\left( \frac{dP}{dQ} \right) dQ$ 2 and empirical risk),
support preservation that imposes strong prior-dominated constraints,
equivalence results between unconstrained and constrained divergence formulations,
tractable dual optimization reducing to scalar non-linear equations or ODEs,
the ability to tailor learning objectives to marginal, tail, or dependency properties using suitable divergence and reference measure choices.

Ongoing research has continued to elaborate efficient algorithms, practical calibrations, and connections to broader function-space learning and modern over-parameterized regimes (Alet et al., 2024, Daunas et al., 2024, Daunas et al., 20 Feb 2025, Daunas et al., 5 Aug 2025).

Key citations: (Daunas et al., 2024, Daunas et al., 20 Feb 2025, Daunas et al., 5 Aug 2025, 0912.0071, Fukuchi et al., 2015, Baharlouei et al., 2023, Daunas et al., 2023, Fröhlich et al., 2022, Coppens et al., 2023, Aminian et al., 2024, Alet et al., 2024)