PAC Indistinguishability: Theory & Algorithms

• PAC Indistinguishability is a framework that generalizes standard PAC learning by focusing on making a predictor indistinguishable from a target using a class of outcome-based distinguishers.
• It leverages metric entropy and dual Minkowski norms to tightly characterize sample complexity, linking classical PAC and agnostic L₁-learning regimes.
• Practical algorithms like Distinguisher-Covering and Multiaccuracy Boost demonstrate how theoretical bounds translate into efficient predictor selection and update procedures.

PAC indistinguishability, also termed no-access Outcome Indistinguishability (OI), generalizes the standard PAC (Probably Approximately Correct) learning paradigm by considering a scenario where the goal is to output a predictor $p$ that cannot be distinguished from a target predictor $p^*$ by a class $D$ of distinguishers, based on the observable outcomes derived from predictions. The distinguishing power of $D$, the interplay between metric entropy and sample complexity, and the duality connections to convex geometry are central to the theory, yielding a framework that interpolates between classical PAC learning and agnostic $L_1$-learning depending on the choice of $D$ (Hu et al., 2022).

1. Formal Definition and Framework

Let $X$ denote an instance space and $\mu\in \Delta_X$ a probability distribution over $X$. A predictor $p: X \to [0,1]$ induces a joint law $p^*$0 on $p^*$1: first sample $p^*$2, then generate $p^*$3. Fix a distinguisher class $p^*$4, possibly randomized.

The distinguishing advantage for $p^*$5 and predictors $p^*$6 is given by

$p^*$7

The maximized distinguishing advantage over $p^*$8 is

$p^*$9

A predictor $D$0 is $D$1-OI to $D$2 under $D$3 if $D$4.

Expressing the distinguisher action as a function $D$5, with $D$6, the distinguishing advantage becomes

$D$7

Thus $D$8 corresponds to the dual Minkowski semi-norm:

$D$9

Variants correspond to realizable vs. agnostic (the ground truth $D$0 in a known class $D$1, or not) and distribution-specific vs. distribution-free (where the learner may or may not adapt to $D$2).

2. Metric Entropy Characterization in the Distribution-Specific Realizable Setting

In the realizable, distribution-specific case, assume $D$3 is the unknown target, $D$4 is the distinguisher class, and $D$5 is fixed.

The central sample complexity measure is the covering number (metric entropy) of $D$6 with respect to the dual Minkowski norm:

$D$7

Lower Bound: Packing arguments yield that any (possibly improper, randomized) learner using $D$8 i.i.d. samples drawn from $D$9 must satisfy:

$L_1$0

Upper Bound: The "Distinguisher-Covering" algorithm computes an approximate $L_1$1-cover of $L_1$2 in the dual norm $L_1$3, empirically estimates $L_1$4 for $L_1$5 in this cover, and selects $L_1$6 minimizing the maximum estimation error. It achieves:

$L_1$7

3. Metric-Entropy Duality: Tight Characterizations

Leveraging the symmetry between covering $L_1$8 by $L_1$9 and $D$0 by $D$1, a metric-entropy duality theorem holds: for any bounded, nonempty $D$2, $D$3, and $D$4,

$D$5

for an absolute constant $D$6. In particular, plugging $D$7, $D$8 yields nearly tight two-sided bounds:

$D$9

This duality connects the sample complexity of PAC indistinguishability to metric entropy duality phenomena in convex geometry. The $X$0 term is essential unless convexity further simplifies the setting (Hu et al., 2022).

4. Distribution-Free Characterization via Fat-Shattering Dimension

In the distribution-free agnostic and realizable settings—typically with $X$1—the sample complexity is governed by the fat-shattering dimension of $X$2, denoted $X$3. For any $X$4, $X$5:

$X$6

This result leverages uniform convergence (via the fat-shattering dimension), and a multiaccuracy boosting algorithm that performs iterative updates: in each round, if there exists $X$7 with sufficient average discrepancy, $X$8 is updated in the direction of $X$9. Each round uses $\mu\in \Delta_X$0 fresh samples and decreases $\mu\in \Delta_X$1 distance by $\mu\in \Delta_X$2. Packing arguments establish the matching lower bound.

5. Separation Between Realizable and Agnostic Regimes

A critical departure from classical PAC theory is the potential for an unbounded separation between realizable and agnostic PAC indistinguishability sample complexity:

Setting	Realizable Sample Complexity	Agnostic Sample Complexity
$\mu\in \Delta_X$3 finite or $\mu\in \Delta_X$4	$\mu\in \Delta_X$5	$\mu\in \Delta_X$6 or $\mu\in \Delta_X$7
$\mu\in \Delta_X$8, $\mu\in \Delta_X$9 arbitrary	$X$0	$X$1

Concretely, for $X$2 differing at one point, realizable OI learning is trivial but agnostic, distribution-free OI requires $X$3 samples. Under restrictions such as $X$4 symmetric convex or $X$5 containing all $X$6 functions (and $X$7 binary), the rates collapse to $X$8 or to the metric-entropy rate.

6. Algorithms for PAC Indistinguishability

Two principal algorithmic approaches realize the aforementioned sample complexity bounds:

• Distinguisher-Covering (Realizable, Distribution-Specific):
  • Cover $X$9 under $p: X \to [0,1]$0.
  • On $p: X \to [0,1]$1 samples $p: X \to [0,1]$2, estimate $p: X \to [0,1]$3 for $p: X \to [0,1]$4 in the cover.
  • Select $p: X \to [0,1]$5 minimizing the worst empirical error across the covering set.
• Multiaccuracy Boost (Distribution-Free):
  • Initialize $p: X \to [0,1]$6.
  • Repeat $p: X \to [0,1]$7 rounds:
  • Draw batch of $p: X \to [0,1]$8 examples.
  • If some $p: X \to [0,1]$9 has discrepancy $p^*$00, update $p^*$01, clipped to $p^*$02.
  • Otherwise, terminate.

Both algorithms yield nearly tight rates matching the theoretical characterizations given by metric entropy and fat-shattering dimension, respectively.

7. Mathematical Constructs and Significance

The theory centralizes two geometric-combinatorial constructs:

• Dual Minkowski Norm: For $p^*$03 and $p^*$04,

$p^*$05

• Metric Entropy (Covering Number):

$p^*$06

These underlie both the upper/lower bounds and duality results. The theory provides the first tight, general characterizations for the number of samples needed to ensure $p^*$07-indistinguishability, providing a continuum of learning-theoretic settings from PAC to fully agnostic $p^*$08-learning by appropriately varying $p^*$09 (Hu et al., 2022).