Parameter Learning of Logic Programs for Symbolic-Statistical Modeling (1106.1797v1)

Abstract: We propose a logical/mathematical framework for statistical parameter learning of parameterized logic programs, i.e. definite clause programs containing probabilistic facts with a parameterized distribution. It extends the traditional least Herbrand model semantics in logic programming to distribution semantics, possible world semantics with a probability distribution which is unconditionally applicable to arbitrary logic programs including ones for HMMs, PCFGs and Bayesian networks. We also propose a new EM algorithm, the graphical EM algorithm, that runs for a class of parameterized logic programs representing sequential decision processes where each decision is exclusive and independent. It runs on a new data structure called support graphs describing the logical relationship between observations and their explanations, and learns parameters by computing inside and outside probability generalized for logic programs. The complexity analysis shows that when combined with OLDT search for all explanations for observations, the graphical EM algorithm, despite its generality, has the same time complexity as existing EM algorithms, i.e. the Baum-Welch algorithm for HMMs, the Inside-Outside algorithm for PCFGs, and the one for singly connected Bayesian networks that have been developed independently in each research field. Learning experiments with PCFGs using two corpora of moderate size indicate that the graphical EM algorithm can significantly outperform the Inside-Outside algorithm.

• The paper introduces a graphical EM algorithm that extends parameter learning in logic programs using support graphs and distribution semantics.
• It demonstrates that, when combined with OLDT search, the algorithm matches the time complexity of methods like Baum-Welch and Inside-Outside.
• Strong PCFG experiments reveal orders of magnitude improvements, underscoring the potential of logic programming for complex probabilistic modeling.

Parameter Learning of Logic Programs for Symbolic-Statistical Modeling

The paper authored by Taisuke Sato and Yoshitaka Kameya introduces a comprehensive framework for statistical parameter learning in parameterized logic programs. It extends the traditional notion of least Herbrand model semantics to a probabilistic interpretation termed distribution semantics. This model provides a robust approach for creating Bayesian networks and other symbolic-statistical frameworks, allowing for logical programs to incorporate probabilistic facts with parameterized distributions.

The authors propose the graphical EM algorithm, an adaptation of the traditional EM algorithm tailored to handle parameterized logic programs. This algorithm processes support graphs, a data structure that describes the logical connections between observations and explanations, enabling efficient learning of model parameters via inside and outside probability computations. The analysis shows that when coupled with the OLDT (Ordered Linear Decision Tree) search strategy for obtaining explanations, the graphical EM algorithm achieves the same time complexity as well-known individual algorithms like Baum-Welch for Hidden Markov Models (HMMs) and Inside-Outside for probabilistic context-free grammars (PCFGs).

The strong numerical results are particularly demonstrated in the learning experiments conducted using PCFGs, where the graphical EM algorithm significantly outperforms the Inside-Outside algorithm, evidenced by orders of magnitude improvements in iteration time for parameter updates. These findings underscore the potential of applying logic programming paradigms to complex, probabilistic models that traditionally relied on statistical methods alone.

Through the comparative analysis presented, the paper outlines scenarios and settings where the graphical EM algorithm retains efficiency, matching specialized models in HMMs, PCFGs, and Bayesian networks. The authors establish that the graphical EM algorithm provides a general-purpose method for parameter learning of sophisticated symbolic-statistical models, a task traditionally faced with significant computational challenges.

The theoretical and practical implications of this work are substantial. By leveraging logic programming’s ability to model complex symbolic-statistical phenomena, this paper extends the terrain of parameter learning, particularly in models where traditional statistical methods pose limitations. The future trajectory, as speculated, includes relaxing some of the stringent assumptions for broader applicability, extending logical frameworks to incorporate continuous random variables, and further exploring stochastic natural language processing models like HPSGs. The burgeoning weakness in current methodologies, such as the probabilistic modeling of recursive structures and learning efficiency improvements, are nascent pathways reflecting both a challenge and an opportunity for exploring more adaptable statistical-learning systems grounded in logic programming.

In conclusion, this research is pivotal in advancing the integration of logic programming with statistical models, providing new vistas for handling probabilistic data in a structured, semantically rich framework, thus encouraging further developments in AI and complex systems modeling.