Empirical Property Optimization (EPO) Oracle

• Empirical Property Optimization (EPO) Oracle is a framework that quantifies and audits properties such as group fairness, prediction error, and robust risk in machine learning.
• It leverages empirical minimization over a strategic class of predictors to decouple the information-theoretic complexity from the computational aspects of optimization.
• The framework establishes rigorous theoretical guarantees, including sample complexity bounds for both finite and infinite predictor classes in PAC auditing setups.

Empirical Property Optimization (EPO) Oracle is a generic framework for statistical auditing of machine learning models, introduced as the core component in property-preserving audits—especially under model updates that may strategically shift the model class. The EPO oracle abstracts property estimation and auditability guarantees for properties such as group fairness, prediction error, and robust risk. It achieves this through empirical minimization over a designated strategic class, separating the information-theoretic complexity of the property from the computational aspects of empirical optimization (Ajarra et al., 9 Jan 2026).

1. Formal Definition and Mathematical Framework

The EPO oracle is defined as follows: Given a property $\mu: F\times\mathcal{P}\to\mathbb{R}$ (e.g., statistical parity, error, robust risk), a class $F$ of predictors $f: X\to Y$, and an i.i.d.\ sample $S=\{(x_i,y_i)\}_{i=1}^m\sim D^m$, along with a loss $\ell_\mu(f,(x,y))\in [0,1]$ such that its empirical average estimates $\mu(f,D)$, the EPO oracle computes

$$(\widehat{f}, \widehat{\mu}) = \mathrm{EPO}_\mu(F, S) := \underset{f\in F}{\arg\min}\; \widehat{E}_S(f, \mu)$$

where

$$\widehat{E}_S(f, \mu) = \frac{1}{m} \sum_{i=1}^m \ell_\mu(f, (x_i, y_i)).$$

For statistical parity (SP), the property-specific formulation is: $$\widehat{E}_S^{SP}(f) = \left|\frac{1}{m_0}\sum_{i: x_i \in X_0} f(x_i) - \frac{1}{m_1}\sum_{i: x_i \in X_1} f(x_i)\right|$$ and

$$\widehat{f} = \arg\min_{f\in F} \widehat{E}_S^{SP}(f), \quad \widehat{\mu} = \widehat{E}_S^{SP}(\widehat{f}).$$

This setup naturally encompasses empirical risk minimization (ERM) as a special case, when $\ell_\mu$ is the usual misclassification indicator.

2. Integration into PAC Auditing

The EPO oracle serves as the algorithmic interface within Probably Approximately Correct (PAC) auditing frameworks. The canonical workflow is as follows:

• An auditor samples $m$ labeled examples $S\sim D^m$.
• A single call to the EPO oracle, $\widehat{f} = \mathrm{EPO}_\mu(F, S)$, is made.
• The output $\widehat{f}$ forms a "prospective" model in $F$ whose empirical property $\widehat{\mu}$ estimates the minimum property over $F$.
• Statistical guarantees are obtained by establishing empirical optimality (the closeness of $\widehat{\mu}$ to the empirical property minimum) and uniform convergence (the closeness of empirical to true property over $F$).

The main theoretical instrument is the Strategic Lemma: Given error tolerance $\epsilon$ and confidence $\delta$, and sample size $m$, if

$$\Pr_S\left[\left|\widehat{E}_S(\widehat{f},\mu) - \min_{f\in F}\widehat{E}_S(f,\mu)\right| > \epsilon/3 \right]\le \delta/2$$

and

$$\Pr_S\left[\sup_{f\in F}\left|\widehat{E}_S(f,\mu) - \mu(f,D)\right| > \epsilon/3\right]\le \delta/2,$$

then the audit is $(\epsilon, \delta)$-weak: $$|\mu(\widehat{f}, D) - \min_{f\in F} \mu(f, D)| \le \epsilon$$ with probability at least $1-\delta$ (Ajarra et al., 9 Jan 2026).

3. Theoretical Guarantees for Group Fairness and the SP-Dimension

For group fairness, especially statistical parity, the SP-dimension (SP(F)) quantifies the combinatorial complexity relevant to auditability. Let $SP(F)$ be defined via

$$\Delta_F^{SP}(S_0, S_1) = \{(A_0, A_1)\mid A_i = c\cap S_i,\; c\in F\},\quad SP(F) = \max_{S=S_0\cup S_1} \log_2 |\Delta_F^{SP}(S_0, S_1)|.$$

$SP(F)$ counts the number of distinct group-wise dichotomies realizable by $F$, and always $SP(F)\le VC(F)$.

Key auditing results for statistical parity:

• If $|F|<\infty$ then $(\epsilon,\delta)$-weak SP-auditing requires

$$m = O\Big(\frac{1}{\epsilon^2} \ln\frac{|F|}{\delta}\Big).$$

• For infinite $F$, necessary sample size is

$$m(F,\epsilon,\delta) = \Omega\left(\frac{SP(F)}{\epsilon^2}\right),$$

and sufficient sample size

$$m(F,\epsilon,\delta) = O\left(\frac{1}{\alpha(1-\alpha)\epsilon^2}\max\left\{\ln\frac{2}{\delta}, 2 SP(F) \ln\frac{e}{\epsilon^2}\right\}\right)$$

where $\alpha,1-\alpha$ are group proportions. Thus, finiteness of $SP(F)$ exactly characterizes auditability.

For strong auditability and prospect ratios, coverage and volume-based ratios are used. For finite $F$,

$$m=O\left(\max\left\{\frac{1}{\epsilon^2}\ln\frac{|F|}{\delta}, \frac{1}{\ln(1/\epsilon)}\ln\frac{|F|}{\delta}\right\}\right),$$

and for infinite $F$,

$$r(\epsilon) = \frac{\mathrm{Vol}\left\{f\in F : |\mu(f)-\mu^*|\le \epsilon\right\}}{\mathrm{Vol}(F)}$$

can be estimated via uniform sampling, with concentration rates specified in detail in Theorem 5 of (Ajarra et al., 9 Jan 2026).

4. Algorithmic Implementation and Computational Aspects

The EPO oracle reduces to a single empirical minimization step, which can be executed using standard optimization methods suitable for the strategic class $F$ (SGD, decision-tree solvers, etc.). For group fairness via statistical parity, the following pseudocode summarizes the procedure:

Inputs: $S=\{(x_i,y_i)\}_{i=1}^m$, groups $X_0, X_1$, class $F$, tolerance $\epsilon$ Algorithm:

1. Partition $S$ into $S_0, S_1$.
2. Define $$\widehat{E}_S(f) = \left| \frac{1}{|S_0|}\sum_{x\in S_0} f(x) - \frac{1}{|S_1|}\sum_{x\in S_1} f(x) \right|$$.
3. Call ERM-oracle: $\widehat{f} := \arg\min_{f\in F} \widehat{E}_S(f)$.
4. Output $\widehat{f}$, $\widehat{\mu} = \widehat{E}_S(\widehat{f})$.

Empirical evaluation of SP costs $O(m)$ per candidate. The optimization step over $F$ dominates overall complexity.

5. Extension Beyond Group Fairness

The EPO oracle is agnostic to the underlying property; replacing the SP loss $\ell_{SP}$ with any $\ell_\mu$ yields an oracle minimizing the empirical instance of $\mu$:

• For prediction error: $\ell(f,(x,y))=\mathbf{1}[f(x)\ne y]$; EPO recovers standard ERM.
• For robust risk: $$\ell(f,(x,y)) = \sup_{z\in U(x)}\mathbf{1}[f(z)\ne y]$$; EPO becomes robust ERM.
• For generalization gap: pairwise losses extend directly.

Analytical validity carries over as long as (i) $\mathbb{E}_S\ell_\mu = \mu_S$, and (ii) the capacity control (e.g., via VC- or Rademacher-dimension) is available for $\{\ell_\mu(f,\cdot): f\in F\}$.

6. Significance and Open Directions

The EPO oracle provides a unifying abstraction, reducing any black-box auditable property to a single step of empirical minimization over a strategic class. Its primary strength lies in orthogonalizing the information complexity of the property (as captured by SP-dimension, VC-dimension, etc.) from the algorithmic complexity (finding $\arg\min_f\widehat{E}_S(f)$). This separation clarifies auditability conditions, optimal sample requirements, and computational feasibility within dynamic or adaptive model settings.

Promising open directions include:

• Interactive or online auditing schemes requiring new sequential complexity measures.
• Architectural integration of audit criteria into learning algorithms.
• Conceptually agnostic audits using dimension- or structure-free approaches.
• Extensions to complex systems such as LLMs or other adaptive infrastructures (Ajarra et al., 9 Jan 2026).