Higher Arity PAC Learning Insights

• Higher Arity PAC Learning is a framework for learning over n-tuple domains, generalizing traditional PAC concepts to relational and hypergraph structures.
• It leverages generalized combinatorial measures and packing lemmas to derive sample complexity bounds and uniform convergence guarantees.
• The approach integrates recursive, exchangeable sampling with algorithmic and recursion-theoretic methods to establish learnability equivalences.

Higher Arity PAC Learning, also referred to as PACₙ Learning, is the study of statistical learning where examples, hypotheses, and target concepts possess arity $n\geq 1$—that is, they are defined on or between $n$-tuples from a domain, rather than on singletons. This framework generalizes classical PAC learnability to settings such as graph, hypergraph, and relational structure learning, where natural problems involve learning functions on $\mathcal{X}^n$ for $n > 1$, and samples are drawn as induced substructures (exchangeable distributions) reflecting combinatorial dependencies. The theory incorporates generalizations of VC dimension (VC$_n$, VCN$_k$), packing lemmas, sample complexity bounds, and regularity methods; and connects these structural characterizations with algorithmic and recursion-theoretic perspectives.

1. Combinatorial Dimensions: VC$_n$ and VCN$_k$

The central structural parameter for Higher Arity PAC Learning is the VC$_n$ (or more generally VCN$_k$) dimension, which extends VC dimension to families of subsets of $n$-fold product spaces. For a class $\mathcal{F}\subseteq V_1\times\dots\times V_n$, its VC$_n$ dimension $d$ is the largest integer such that there exists a $d$-box $A=A_1\times \dots \times A_n$, $|A_i|=d$, for which every subset $A'\subseteq A$ occurs as $A\cap S$ for some $S\in \mathcal{F}$. Formally,

$$\forall\,A'\subseteq A,\quad\exists\,S\in\mathcal{F}\;\text{such that}\;A'=A\cap S.$$

In function classes $H\subseteq Y^{X^k}$, the VCN$_k$ dimension is defined by slicing at fixed $(k-1)$-tuples; for each $x\in X^{k-1}$, one examines the induced class $H(x)$ of functions on the remaining coordinate, then VCN$_k(H)=\sup_{x\in X^{k-1}} \operatorname{Nat}(H(x))$ (where $\operatorname{Nat}$ is the Natarajan dimension if $Y$ is non-binary).

This generalization preserves critical connections between combinatorial shattering and learnability in higher-arity scenarios, providing a necessary and sufficient condition for PAC$_n$ learnability in terms of finiteness of the VC$_n$/VCN$_k$ dimension (Chernikov et al., 2 Oct 2025, Coregliano et al., 21 May 2025).

2. Generalized Haussler Packing and Covering Properties

In the unary ($n=1$) setting, the Haussler packing lemma asserts that classes of finite VC dimension may be covered (in sample/measure approximation) by a bounded number of representatives. The higher-arity setting requires refinements: given a class $\mathcal{F}$ of subsets of $V_1\times\dots\times V_n$ with VC$_n$ dimension $d$, for each product probability measure $\mu_1\otimes\dots\otimes\mu_n$, there exists a finite family $\{S_i\}_{i=1}^N$ such that every $S\in\mathcal{F}$ can be approximated (in measure) by a Boolean combination of the $S_i$ and lower-arity fibers. Quantitatively, for all $S$, there exists $D$ such that

$$\mu_1\otimes\dots\otimes\mu_n\left(S \, \Delta \, D \right) \le \varepsilon,$$

with $N=N(n,d,\varepsilon)$ bounding the complexity. This lemma is crucial in establishing uniform convergence, agnostic and non-agnostic sample complexity bounds, and derandomization techniques in PAC$_n$ learning (Chernikov et al., 2 Oct 2025, Coregliano et al., 21 May 2025).

3. Characterization and Learnability Equivalences

A comprehensive characterization now exists for PAC$_n$ learning in product spaces, particularly:

• $\mathcal{F}$ has finite VC$_n$ dimension.
• $\mathcal{F}$ satisfies a generalized Haussler packing property.
• $\mathcal{F}$ exhibits uniform convergence (for both non-partite and partite sampling schemes).
• $\mathcal{F}$ is agnostic and non-agnostic PAC$_n$ learnable.

These conditions are logically equivalent and imply the existence of efficient learning algorithms whose sample complexity depends polynomially (or nearly so) on the combinatorial dimension $d$ and accuracy/confidence parameters $\varepsilon,\delta$ (Coregliano et al., 21 May 2025, Chernikov et al., 2 Oct 2025). For example, in Boolean classes,

$$m(\varepsilon, \delta) = O\left(\frac{d\log\frac{d}{\varepsilon}+\log\frac{1}{\delta}}{\varepsilon^2}\right)$$

generalizes to higher arity with minor modifications dictated by the structure induced by $n$-tuples.

4. Sampling Models and Exchangeability

The classical PAC model relies on i.i.d. samples from a measure $\mu$ over $X$. Higher arity PAC theory modifies this: samples are drawn as tuples (e.g., edges in graphs, hyperedges, relations), producing exchangeable distributions. The product measure $\mu^{n}$ on $X^n$ governs the sampling, and exchangeability reflects symmetry (e.g., all pairs of vertices are treated equivalently).

This structured sampling mechanism is central in learning induced substructures in graphs, hypergraphs, and logic models, enabling generalization to statistical learning where independence does not strictly hold. The regularity methods developed for higher arity settings establish that slice-wise regular partitions with small exceptional sets are possible under bounded VC$_n$ dimension (Chernikov et al., 2 Oct 2025).

5. Sample Complexity and Algorithmic Methods

Bounds for sample complexity and covering numbers in PAC$_n$ learning closely resemble their unary analogues, with dependence on the combinatorial dimension. The optimal sample complexity for binary PAC learning—$$m(\varepsilon,\delta) = O((1/\varepsilon)(d+\ln(1/\delta)))$$—carries over to higher arity with aggregation and voting schemes appropriately generalized (plurality, multi-vote) (Hanneke, 2015). Recursive subsampling and majority/plurality voting across base learners can be adapted to $n$-ary or relational outputs, with ensemble methods yielding robust guarantees.

For agnostic learning of statistical (distributional) function classes derived from base classes via expectation/randomization, explicit sample complexity bounds can be established in terms of the fat-shattering or VC dimension of the base class (Anderson et al., 1 Apr 2025). In realizable learning, fundamental limitations exist: counterexamples demonstrate that the mere realizability of the base class does not guarantee realizable learnability of the derived statistical class.

6. Recursion-Theoretic and Arithmetic Hierarchy Complexity

From a recursion-theoretic perspective, the characterization of learnability (finite VC dimension) for effective concept classes is precisely at the $\Sigma^0_3$ (for learnable classes) or $m$-complete $\Pi^0_3$ (for unlearnable classes) level within the arithmetic hierarchy (Calvert, 2014). This applies uniformly to higher arity PAC learning: the shattering conditions and the associated combinatorial definitions generalize to tuples, and the computational complexity of deciding learnability is equivalently intricate in the $n$-ary setting.

Formally, the condition for infinite VC dimension (and hence non-learnability) is expressible by polyquantifier formulas (e.g., $$\forall n\in\mathbb{N}\;\exists x_1,\dots,x_n \;\forall S\subseteq[n]\;\exists c\in \mathcal{C}\;\ldots$$), which also hold in higher arity scenarios when “shattering” refers to sets of n-tuples.

7. Connections to Model Theory, Relational Learning, and Practical Implications

Methods from model theory, especially randomization of structures (Keisler, Ben Yaacov, Towsner), provide a deep structural underpinning for higher arity PAC theory (Anderson et al., 1 Apr 2025). Randomization techniques and slice-wise regularity lemmas support the approximate partitioning of complex hypergraph relations. Practical algorithms exploit ensemble voting, recursive data partitioning, and exchangeability to learn relational models across diverse domains.

Partial concept classes, which model functions undefined on portions of the space, extend the scope of higher arity PAC learning to scenarios with data lying on submanifolds, margin conditions, and other realistic constraints (Alon et al., 2021). These settings reveal failures of sample compression conjectures and the limits of ERM in learning partial or relational functions.

Summary Table: Key Structural Parallels (High-Arity vs Unary PAC)

Dimension	Classical (Unary)	Higher Arity ($n$-ary, $k$-ary)
VC dimension	VC	VC$_n$, VCN$_k$
Packing lemma	Haussler covering	Generalized Haussler packing (boxes, cylinders, fibers)
Learning equivalence	Finite VC $\iff$ PAC	Finite VC$_n$/VCN$_k$ $\iff$ PAC$_n$
Regularity lemma	Graph partitions	Slice-wise hypergraph regularity
Recursion-theoretic	$\Sigma^0_3$/$\Pi^0_3$	Same, shattering with $n$-tuples

The integration of combinatorial, algorithmic, and logical perspectives in higher arity PAC learning yields a mature structural theory. It fully characterizes learnability for complex relational systems and explains effective methods across statistical, agnostic, and non-agnostic regimes, with precise sample complexity and computational bounds in terms of VC$_n$/VCN$_k$ dimensions and packing properties. The field continues to connect deep structural regularity notions with practical algorithms and program complexity, illuminating learning in high-dimensional, structured domains.