Learning Concepts Definable in First-Order Logic with Counting (1909.03820v2)

Abstract: We study Boolean classification problems over relational background structures in the logical framework introduced by Grohe and Tur\'an (TOCS 2004). It is known (Grohe and Ritzert, LICS 2017) that classifiers definable in first-order logic over structures of polylogarithmic degree can be learned in sublinear time, where the degree of the structure and the running time are measured in terms of the size of the structure. We generalise the results to the first-order logic with counting FOCN, which was introduced by Kuske and Schweikardt (LICS 2017) as an expressive logic generalising various other counting logics. Specifically, we prove that classifiers definable in FOCN over classes of structures of polylogarithmic degree can be consistently learned in sublinear time. This can be seen as a first step towards extending the learning framework to include numerical aspects of machine learning. We extend the result to agnostic probably approximately correct (PAC) learning for classes of structures of degree at most $(\log \log n)^c$ for some constant $c$. Moreover, we show that bounding the degree is crucial to obtain sublinear-time learning algorithms. That is, we prove that, for structures of unbounded degree, learning is not possible in sublinear time, even for classifiers definable in plain first-order logic.

• The paper extends learnability from first-order logic to first-order logic with counting, broadening expressive classification methods in relational structures.
• It introduces sublinear time algorithms that learn with polylogarithmic degree bounds by employing Hanf locality techniques.
• The study establishes PAC learning frameworks for FOCN and proves negative results for unbounded degree structures.

Learning Concepts Definable in First-Order Logic with Counting

The paper under discussion explores the learnability of Boolean classification problems in the field of relational background structures. Specifically, it extends the foundational work in first-order logic to encompass first-order logic with counting (FOCN), offering a comprehensive exploration of learning scenarios and efficiency in a logical context.

Overview of Contributions

This research builds upon the framework established by Grohe and Turán for learning over relational structures using first-order logic. It highlights a broadened approach by incorporating FOCN, a logic introduced by Kuske and Schweikardt. The inclusion of FOCN is a significant step as it captures numerical aspects akin to SQL's COUNT operation, thus facilitating more expressive hypotheses in logical classifications.

The paper's core contributions are multifold:

1. Extension to FOCN: The authors have extended the concept of learnability, initially established for first-order logic, to first-order logic with counting. This extension is non-trivial due to FOCN's numerically expressive nature, which surpasses the descriptive capabilities of basic first-order logic.
2. Learning with Degree Boundaries: The research asserts that learnable hypotheses are consistent even on classes of structures with polylogarithmic degrees. A notable achievement here is the demonstration of learning possible not only under bounded degree restrictions but also when the degree constraints are polylogarithmic relative to the structure's size.
3. Algorithmic Results: The consistent-learning algorithm devised runs in sublinear time for structures bounded by polylogarithmic degree. This efficiency is achieved by employing techniques adapted from Hanf locality to manage spheres of influence around elements in the structure, crucial for efficient hypothesis evaluation.
4. PAC Learning: The authors extend their results into the agnostic probably approximately correct (PAC) learning framework, providing bounds on sample complexity to achieve PAC results. This extends the applicability and robustness of their framework against inaccurate or non-representative sample data.
5. Negative Results for Unbounded Degrees: The work includes proofs showing the infeasibility of sublinear time learning for structures without bounded degree. Hence, it underscores the essential role of structural degree constraints for efficient learnability in logical frameworks.

Implications and Future Directions

The implications of these results are profound for both theoretical explorations and practical implementations of machine learning within logical and structured data domains. By providing efficient algorithms for learning in contexts extending beyond conventional first-order logic, this paper paves the way for more nuanced data analysis scenarios, especially where numerical aggregations play a pivotal role.

Theoretically, the findings underscore the potential for logical frameworks to embrace other aggregation mechanisms beyond counting, perhaps considering logics like FOWA (First-Order Logic with Weight Aggregation) that account for varied weights and measures.

The open questions poised by the paper involve exploring the boundaries of learning in yet more expressive logics and considering the implications of different aggregation forms within a learning context. If successful, such endeavors could produce new breakthroughs in logic-guided machine learning models.

In sum, this paper offers a significant stride in understanding how logical expressiveness intersects with computational learnability, pushing the frontier towards more powerful, logically-underpinned machine learning algorithms.