Comparing Bayesian Network Classifiers (1301.6684v1)

Abstract: In this paper, we empirically evaluate algorithms for learning four types of Bayesian network (BN) classifiers - Naive-Bayes, tree augmented Naive-Bayes, BN augmented Naive-Bayes and general BNs, where the latter two are learned using two variants of a conditional-independence (CI) based BN-learning algorithm. Experimental results show the obtained classifiers, learned using the CI based algorithms, are competitive with (or superior to) the best known classifiers, based on both Bayesian networks and other formalisms; and that the computational time for learning and using these classifiers is relatively small. Moreover, these results also suggest a way to learn yet more effective classifiers; we demonstrate empirically that this new algorithm does work as expected. Collectively, these results argue that BN classifiers deserve more attention in machine learning and data mining communities.

• The paper demonstrates that CI-based BN classifiers can outperform traditional Naïve-Bayes and TAN on large datasets.
• It evaluates four BN classifier variants using mutual information tests and five-fold cross-validation for robust empirical analysis.
• The study highlights practical benefits like efficient feature selection, sensitivity to threshold tuning, and scalability in large-scale applications.

Analyzing Bayesian Network Classifiers

Bayesian Networks (BNs) have gained increased attention for their ability to represent probabilistic models and perform inference under uncertainty. The paper by Cheng and Greiner provides an empirical evaluation of algorithms designed to train four types of BN classifiers: Naïve-Bayes, Tree Augmented Naïve-Bayes (TAN), BN Augmented Naïve-Bayes (BAN), and General BNs (GBNs). The authors utilize conditional-independence (CI) based algorithms to construct these classifiers and compare their performance with other established classifiers.

Classifier Types and Learning Algorithms

1. Naïve-Bayes (NB): A simple BN structure assuming feature independence given the class label. It is computationally efficient and surprisingly effective, despite the unrealistic independence assumption.
2. Tree Augmented Naïve-Bayes (TAN): This extends NB by allowing dependencies between features in the form of a tree structure, enhancing its representational capabilities while maintaining computational efficiency.
3. BN Augmented Naïve-Bayes (BAN): BANs allow for more general dependencies among features, going beyond tree structures to employ arbitrary graphs, providing more flexibility.
4. General BNs (GBN): These treat the class node without special privileges, enabling the learning of a full BN structure. The model can capture complex dependencies but risks overfitting with smaller datasets.

Methodology and Evaluation

The authors use eight datasets from the UCI Machine Learning Repository, focusing on datasets with large numbers of examples and few continuous features. The empirical analysis comprises learning and testing classifiers using five-fold cross-validation for datasets without predefined training/test splits. The algorithms use the PowerConstructor 2.0 program for learning BN structures, utilizing mutual information tests to identify dependencies among attributes.

Experimental Results

The experimental results suggest that CI-based BN classifiers (GBN and BAN) are competitive with, and often superior to, Naïve-Bayes and TAN, particularly on larger datasets. For instance, BAN achieved a perfect classification on the 'Mushroom' dataset. However, for some datasets like 'Nursery' and 'Car', GBN's performance was suboptimal due to its reduction to a selective Naïve-Bayes, which failed to capture weak dependencies.

The paper also examined the computational efficiency of these algorithms. Learning times were generally short, with BAN learning on the largest dataset (Adult) taking approximately nine minutes, demonstrating practical efficiency.

Implications and Future Work

The analysis demonstrates several implications for the use of BN classifiers:

• Feature Selection: GBN's potential to use the Markov blanket as a natural feature subset provides an efficient mechanism for feature selection.
• Threshold Sensitivity: The performance of unrestricted BN classifiers is sensitive to threshold settings in CI tests. Proper threshold tuning, as demonstrated by the proposed wrapper algorithm, can significantly enhance performance.
• Scalability: The paper assures that even sophisticated BN classifiers can be trained efficiently, making them viable for large-scale applications.

Parting Thoughts

By empirically demonstrating the strengths of CI-based learning algorithms for BNs, this paper advocates for their broader adoption in machine learning and data mining applications. The proposed methodologies for automatic threshold tuning and classifier wrapping offer paths for further development, promising enhanced accuracy and robustness of BN classifiers in future AI research endeavors.