Interpretable multiclass classification by MDL-based rule lists (1905.00328v2)

Abstract: Interpretable classifiers have recently witnessed an increase in attention from the data mining community because they are inherently easier to understand and explain than their more complex counterparts. Examples of interpretable classification models include decision trees, rule sets, and rule lists. Learning such models often involves optimizing hyperparameters, which typically requires substantial amounts of data and may result in relatively large models. In this paper, we consider the problem of learning compact yet accurate probabilistic rule lists for multiclass classification. Specifically, we propose a novel formalization based on probabilistic rule lists and the minimum description length (MDL) principle. This results in virtually parameter-free model selection that naturally allows to trade-off model complexity with goodness of fit, by which overfitting and the need for hyperparameter tuning are effectively avoided. Finally, we introduce the Classy algorithm, which greedily finds rule lists according to the proposed criterion. We empirically demonstrate that Classy selects small probabilistic rule lists that outperform state-of-the-art classifiers when it comes to the combination of predictive performance and interpretability. We show that Classy is insensitive to its only parameter, i.e., the candidate set, and that compression on the training set correlates with classification performance, validating our MDL-based selection criterion.

• The paper introduces Classy, an algorithm that builds interpretable multiclass classification models using MDL-based rule lists to balance accuracy and model complexity.
• Classy leverages the Minimum Description Length principle and a greedy search strategy to select compact, predictive rule lists without extensive hyperparameter tuning.
• Empirical validation shows Classy produces more compact, interpretable models that perform comparably to or better than state-of-the-art classifiers on diverse datasets.

Interpretable Multiclass Classification via MDL-Based Rule Lists

The paper "Interpretable multiclass classification by MDL-based rule lists" addresses the rising demand for transparent and interpretable machine learning models. The authors present a methodology for building compact yet interpretable models using rule lists and the Minimum Description Length (MDL) principle. This aligns with the increasing need for models that are not only predictive but also understandable, particularly in high-stakes domains like healthcare and social applications.

Key Contributions

The authors introduce Classy, an algorithm that constructs rule lists for multiclass classification. The design of Classy is guided by the MDL principle, ensuring that model complexity is judiciously balanced with predictive accuracy. This approach effectively mitigates overfitting and obviates the need for extensive hyperparameter tuning. Classy's primary advantage lies in its practicable balance of interpretability and performance, which is particularly beneficial in environments where model transparency is crucial.

Methodology

The authors leverage probabilistic rule lists, where each rule consists of a pattern (antecedent) and an associated probability distribution of class labels (consequent). The MDL principle is employed to formalize the selection of these rule lists by minimizing the description length of the data given the model and the model itself. This parameter-free approach selects models based on their ability to compress the data, an effective proxy for model generalizability.

For practical implementation, Classy utilizes a greedy search strategy, iteratively adding rules that yield the highest normalized compression gain. The algorithm employs frequent pattern mining to generate potential rule candidates, enabling Classy to handle large candidate sets without significant degradation in performance.

Empirical Validation

The authors demonstrate the effectiveness of Classy across 17 diverse datasets. The system consistently outperforms or matches other state-of-the-art classifiers like CART, C5.0, JRip, and SBRL in terms of AUC, particularly in multiclass settings. Classy's models are notably more compact, having fewer rules and conditions, which enhances their interpretability. The authors show a strong correlation between better data compression and higher predictive accuracy, validating the MDL-based approach.

Implications and Future Directions

The research provides a significant step towards deployable interpretable models, reducing the dependency on rigorous hyperparameter tuning and making them accessible in real-time applications. Future work could extend the current rule-based methodology into other types of data and applications, such as continuous variables or regression tasks. A promising line of investigation is the development of hybrid search methods combining the completeness of optimal strategies and the efficiency of greedy approaches for learning rule lists.

Conclusion

This paper contributes to the ongoing discourse on interpretable machine learning by presenting a robust, theoretically substantiated method for generating interpretable models that do not compromise on predictive strength. Through the Classy algorithm and its MDL-based framework, the research offers practical tools for understanding complex data-driven decisions, a crucial capability in this age of widespread machine learning application.