On the Power of Interactive Proofs for Learning (2404.08158v1)

Abstract: We continue the study of doubly-efficient proof systems for verifying agnostic PAC learning, for which we obtain the following results. - We construct an interactive protocol for learning the $t$ largest Fourier characters of a given function $f \colon {0,1}^n \to {0,1}$ up to an arbitrarily small error, wherein the verifier uses $\mathsf{poly}(t)$ random examples. This improves upon the Interactive Goldreich-Levin protocol of Goldwasser, Rothblum, Shafer, and Yehudayoff (ITCS 2021) whose sample complexity is $\mathsf{poly}(t,n)$. - For agnostically learning the class $\mathsf{AC}^0[2]$ under the uniform distribution, we build on the work of Carmosino, Impagliazzo, Kabanets, and Kolokolova (APPROX/RANDOM 2017) and design an interactive protocol, where given a function $f \colon {0,1}^n \to {0,1}$, the verifier learns the closest hypothesis up to $\mathsf{polylog}(n)$ multiplicative factor, using quasi-polynomially many random examples. In contrast, this class has been notoriously resistant even for constructing realisable learners (without a prover) using random examples. - For agnostically learning $k$-juntas under the uniform distribution, we obtain an interactive protocol, where the verifier uses $O(2^k)$ random examples to a given function $f \colon {0,1}^n \to {0,1}$. Crucially, the sample complexity of the verifier is independent of $n$. We also show that if we do not insist on doubly-efficient proof systems, then the model becomes trivial. Specifically, we show a protocol for an arbitrary class $\mathcal{C}$ of Boolean functions in the distribution-free setting, where the verifier uses $O(1)$ labeled examples to learn $f$.

• The paper introduces interactive proofs that significantly reduce sample complexity for verifying agnostic PAC learning of diverse Boolean function classes.
• It details a novel protocol for efficiently identifying heavy Fourier characters, offering doubly-efficient performance over previous methods.
• The framework extends verification to AC0[2] circuits and k-juntas, paving the way for robust learning techniques in computationally complex settings.

Interactive Proofs for Verifying Agnostic PAC Learning of Boolean Functions

Overview

The paper of interactive proof (IP) systems for verifying the results of computational tasks has profoundly impacted theoretical computer science. Rooted in this tradition, a paper extends the exploration of IPs into the field of agnostic Probably Approximately Correct (PAC) learning of Boolean functions. This work particularly addresses verifying the learning of significant classes such as large Fourier coefficients, $AC^0[2]$, $k$-juntas, and general circuit classes, showcasing the power of interactive proofs in a PAC learning context.

Learning Heavy Fourier Characters

Fourier analysis is a powerful tool in understanding Boolean function complexity. Identifying heavy Fourier coefficients (characters) plays a critical role in algorithms across learning theory, coding, and cryptography. The paper introduces a doubly-efficient interactive protocol for finding the $t$ heaviest Fourier characters of a Boolean function $f: \{0,1\}^n \to \{0,1\}$. The protocol requires $poly(t/)$ random examples, a quantitative improvement over previous works, lending a sample-efficient methodology for learning heavy Fourier characters.

Verifying Learning $AC^0[2]$ Circuits

$AC^0[2]$ represents a class of constant-depth polynomial-size circuits including AND, OR, NOT, and XOR gates. The paper builds an agnostic PAC-verifier for $AC^0[2]$, projecting a scenario where collision-resistant hashing can be bypassed in accessing the closest hypothesis in a PAC-verification model. Through an interactive protocol, it demonstrates that a quasi-polynomial number of random examples sufficiently estimate the distance between a given function and its closest hypothesis in $AC^0[2]$, culminating in a doubly efficient learning framework.

Agnostic Verification for $k$-Juntas

$k$-juntas, functions depending on at most $k$ of their input bits, have posed significant challenges in the field of learning theory. The paper presents an interactive protocol for verifying the learning of $k$-juntas under the uniform distribution using $O(2^k)$ random examples, independent of $n$. This efficiency promotes a practical approach towards learning $k$-juntas, contributing to understanding their complex nature.

Traditional Learning Transformed

The paper illustrates that constructing a novel interactive protocol allows for learning arbitrary classes of Boolean functions with trivial sample complexity in the distribution-free setting. By delegating the learning task to an unbounded prover, it achieves distribution-free learning of $P/poly$ with $O(1/)$ labeled examples—a breakthrough that simplifies the learning process for any class of Boolean functions.

Implications and Future Directions

This paper's implications are vast, introducing a framework that could redefine agnostic PAC learning through interactive proofs. Beyond the academic curiosity, it paves the way for practical verification systems in machine learning, where computational resources and data access are limited. Speculating on future developments, one could foresee the evolution of more sophisticated protocols for broader classes and the exploration of IP models in other learning paradigms.

Conclusion

Interactive proofs for learning represent a frontier in the overlap of computational complexity, machine learning, and cryptography. This paper leverages the structural properties of Boolean functions to efficiently verify agnostic PAC learning tasks, marking significant progress in the theory and possibly the practice of machine learning.