Tight Bounds for Learning Polyhedra with a Margin

Abstract: We give an algorithm for PAC learning intersections of $k$ halfspaces with a $ρ$ margin to within error $\varepsilon$ that runs in time $$\textsf{poly}(k, \varepsilon^{-1}, ρ^{-1}) \cdot \exp \left(O(\sqrt{n \log(1/ρ) \log k})\right)$$. Notably, this improves on prior work which had an exponential dependence on either $k$ or $ρ<sup>{-1}$ and matches known cryptographic and Statistical Query lower bounds up to the logarithmic factors in $k$ and $ρ$ in the exponent. Our learning algorithm extends to the more general setting when we are only promised that most points have distance at least $ρ$ from the boundary of the polyhedron, making it applicable to continuous distributions as well.

• The paper presents an algorithm with runtime 2^(O(sqrt(n log(1/ρ) log(kε⁻¹)))) that overcomes previous exponential-time limits for learning polyhedral intersections.
• It leverages a novel sampling-based weak learning subroutine combined with region covering boosting to maintain low error in agnostic settings.
• The work matches established SQ and cryptographic lower bounds, offering tight complexity tradeoffs for high-dimensional geometric learning.

Tight Bounds for PAC Learning Intersections of Halfspaces with a Margin

Introduction and Motivation

The paper "Tight Bounds for Learning Polyhedra with a Margin" (2604.14614) tackles the longstanding problem of PAC learning intersections of $k$ halfspaces in $\mathbb{R}^n$ with margin $\rho$, focusing on the case where $k$ and $\rho^{-1}$ are polynomially bounded in $n$. Intersections of halfspaces encapsulate key classes such as polyhedra, DNFs, and generalize the well-studied single halfspace (LTF) setting. Unlike the single halfspace case—where efficient (polynomial-time) algorithms are known under minimal assumptions—learning their intersection, even with generous geometric separation, resisted efficient learning guarantees due to known exponential lower bounds from both Statistical Query (SQ) and cryptography-based frameworks.

Previous algorithms for this problem suffered from exponential dependence either on $k$ or $\rho^{-1}$, resulting in exponential-time even for the regime of $k = \text{poly}(n)$ and $\rho = 1/\text{poly}(n)$. The main contribution of this work is an essentially tight algorithm whose runtime is $\mathbb{R}^n$0, overcoming previous limitations and matching known lower bounds up to logarithmic factors.

Problem Formalization

The main object of study is the class of intersections of $\mathbb{R}^n$1 halfspaces

$\mathbb{R}^n$2

where $\mathbb{R}^n$3, each $\mathbb{R}^n$4, and corresponding thresholds $\mathbb{R}^n$5. The learning setting is agnostic, allowing $\mathbb{R}^n$6 to be any distribution over $\mathbb{R}^n$7. The margin $\mathbb{R}^n$8 specifies that every point $\mathbb{R}^n$9 labeled negative lies at least $\rho$0 away from all boundary hyperplanes (see Definition 1 in the paper).

Main Algorithmic Contributions

The paper presents an algorithm with the following key performance guarantee:

Theorem (Informal). There is an algorithm for PAC learning intersections of $\rho$1 halfspaces with margin $\rho$2 to error $\rho$3 over arbitrary distributions, running in

$\rho$4

time. The guarantee holds even under a soft margin assumption (i.e., if all but an $\rho$5 fraction of points have distance at least $\rho$6 to the polyhedron's boundary).

This runtime is strictly subexponential in $\rho$7 for polynomial $\rho$8 and $\rho$9. Notably, the dependence on $k$0 and $k$1 is moved inside the square root in the exponent, outperforming prior work where these appeared linearly or polynomially in the exponent.

Technical Overview

The core algorithm employs a novel weak learning subroutine combined with boosting by region covering. Unlike the classical paradigm, which leverages polynomial threshold function approximations (PTF) and faces degree-based exponential lower bounds, their method constructs a distribution over halfspaces by sampling from a convex body determined by classifying random positive and negative samples correctly.

The key insight is to find a halfspace $k$2 such that:

• The region $k$3 contains a nontrivial (exponentially small in $k$4 but adequate for boosting) fraction of negative points, but few positives.
• $k$5 is significantly biased towards $k$6 on the region defined by $k$7.

The polyhedral region is defined by taking random samples of positive and negative points and considering all halfspaces that agree with these on all samples. By careful volume analysis, most halfspaces in this region are shown to have the required property, provided the sample sizes are sufficiently large—crucially tuned so that the resulting advantage is roughly $k$8. This suffices for boosting, ultimately yielding a strong hypothesis.

When relaxing from the hard margin setting to a soft margin (where an $k$9 fraction of the probability mass can be arbitrarily close to the boundary), the authors show that their algorithm maintains identical performance guarantees for almost all points, only accruing an additive error of $\rho^{-1}$0. This soft margin scenario captures continuous distributions, including mixtures of Gaussians for which classical hard margin-based methods fail.

Figure 1: The soft margin setting (left) permits an $\rho^{-1}$1 fraction of negative points within $\rho^{-1}$2 of the positive region; the construction of the good region (right) for random positive and negative samples is central to the algorithm.

Lower Bounds & Optimality

The algorithm matches (up to logarithmic factors) both:

• The $\rho^{-1}$3 SQ lower bound for learning the intersection of $\rho^{-1}$4 low-weight halfspaces with inverse polynomial margin [klivans2007unconditional].
• The cryptographic lower bound based on the hardness of SVP in lattices [klivans2009cryptographic, tiegel2024improved], dictating the (conditional) best-possible efficiency for algorithms not exploiting non-SQ structure.

Proof and Analysis Highlights

The theoretical analysis relies on geometric probability, volumetric estimates, and VC theory:

• Sampling-based construction ensures with high probability that any sampled halfspace correctly classifies nearly all the positive (negative) samples.
• Good halfspaces dominate in measure, while bad regions shrink exponentially fast with the number of negative samples.
• The overall error is controlled via a region-based boosting procedure, which recursively restricts the instance space while accumulating minimal positive classification error at each step.

Notably, the covering-based boosting carefully ensures the cumulative hypothesis is a proper intersection of halfspaces, which is necessary for distribution-independent PAC learning with a proper hypothesis class.

Implications and Future Directions

This work closes the gap for learning polyhedral classes in the high-dimensional margin regime:

• Establishes essentially tight complexity tradeoffs for the intersection class in the (soft/hard) margin scenario.
• Provides constructive learning methods in instances previously seen as hard on both statistical and cryptographic grounds.
• The sampling-from-convex-region method and covering-based boosting strategy may find application in other geometric learning contexts where analogous sample complexity and volumetric bottlenecks appear.

Further investigation may address extensions to richer concept classes (e.g., unions of polyhedra), agnostic settings with arbitrary label noise, or tightening the logarithmic factor dependencies under various distributional constraints.

Conclusion

The paper provides an optimal (up to logarithmic factors) algorithm for PAC learning intersections of $\rho^{-1}$5 halfspaces with margin, generalizing to soft margins and arbitrary input distributions. The combination of sample-based polytope construction, geometric analysis, and region-based boosting achieves provable learning efficiency matching both information-theoretic statistical query and cryptographic lower bounds for this class, resolving a major open question in high-dimensional computational learning theory.