Most Influential Subset Selection: Challenges, Promises, and Beyond (2409.18153v2)

Abstract: How can we attribute the behaviors of machine learning models to their training data? While the classic influence function sheds light on the impact of individual samples, it often fails to capture the more complex and pronounced collective influence of a set of samples. To tackle this challenge, we study the Most Influential Subset Selection (MISS) problem, which aims to identify a subset of training samples with the greatest collective influence. We conduct a comprehensive analysis of the prevailing approaches in MISS, elucidating their strengths and weaknesses. Our findings reveal that influence-based greedy heuristics, a dominant class of algorithms in MISS, can provably fail even in linear regression. We delineate the failure modes, including the errors of influence function and the non-additive structure of the collective influence. Conversely, we demonstrate that an adaptive version of these heuristics which applies them iteratively, can effectively capture the interactions among samples and thus partially address the issues. Experiments on real-world datasets corroborate these theoretical findings and further demonstrate that the merit of adaptivity can extend to more complex scenarios such as classification tasks and non-linear neural networks. We conclude our analysis by emphasizing the inherent trade-off between performance and computational efficiency, questioning the use of additive metrics such as the Linear Datamodeling Score, and offering a range of discussions.

• The paper examines the limitations of static influence-based greedy heuristics by revealing their failure to accurately assess collective influence, especially in linear regression scenarios.
• It introduces an adaptive greedy algorithm that iteratively refits the model to better capture the interactions among training samples in both linear and nonlinear tasks.
• The study proves theoretical conditions for heuristic failures and discusses the trade-offs between computational efficiency and robust, accurate influential subset selection.

Most Influential Subset Selection: Challenges, Promises, and Beyond

The paper "Most Influential Subset Selection: Challenges, Promises, and Beyond" by Yuzheng Hu, Pingbang Hu, Han Zhao, and Jiaqi W. Ma addresses the problem of attributing the behavior of machine learning models to specific subsets of training samples. While the influence function has been utilized for gauging the impact of individual data points, the focus of this work is on understanding the collective influence of subsets in order to identify the most influential subset of data points—Most Influential Subset Selection (MISS).

Contributions

The paper undertakes a rigorous analysis of prevailing approaches for MISS, evaluating their strengths and weaknesses, and highlighting the intrinsic challenges. The key contributions discussed in the paper are as follows:

1. Evaluation of Influence-Based Greedy Heuristics: The paper critically examines influence-based greedy heuristics, methods that calculate static scores for each training sample and perform greedy selection based on these scores. The analysis reveals that these heuristics can fail even in simple scenarios such as linear regression due to errors in influence function and the non-additive nature of collective influence. Specifically, individual influence estimates can be inaccurate and misleading for samples with high leverage scores, leading to suboptimal subset selection.
2. Adaptive Greedy Algorithm: The authors propose a refined adaptive greedy algorithm. Unlike static greedy methods, this approach iteratively updates the influence scores by refitting the model after each selection. This dynamic nature allows the algorithm to better capture interactions among samples and addresses the issues identified in static greedy methods. The experimental results demonstrate that the adaptive algorithm effectively extends the benefits of adaptivity to more complex scenarios like classification tasks and non-linear models.
3. Theoretical Findings and Empirical Validation: The paper proves the conditions under which influence-based greedy heuristics fail. For example, the influence function does not account for leverage scores in linear models, leading to incorrect selection of influential samples. Furthermore, the non-additive structure in collective influence, characterized by amplification and cancellation effects, can cause the failure of leverage-adjusted greedy selection (LAGS). The paper provides theoretical proofs, supported by simulations and real-world empirical data, showing the superior performance of the adaptive greedy algorithm.
4. Discussion on Trade-offs: The analysis concludes with a discussion on the trade-off between computational efficiency and performance. While influence-based greeds are computationally efficient, their lack of robustness and accuracy prompts the need for adaptivity or higher-order approximations. The adaptive greedy algorithm, though more computationally intensive, is shown to be more effective in practice.

Implications and Future Directions

The practical implications of this work are significant for domains that rely on interpretable and robust machine learning models, such as healthcare, economics, and public policy. By identifying the subsets of data that have the most significant influence on model predictions, practitioners can enhance data cleaning, model debugging, and interpretability efforts. This, in turn, fosters trust in machine learning models.

From a theoretical standpoint, this paper underscores the limitations of the commonly used influence function and static greedy heuristics, pushing the boundary towards more refined and reliable methods. The proposed adaptive greedy algorithm represents an important step forward in understanding the collective influence of data subsets. Yet, the paper also raises crucial questions about the need for robust methods that can handle higher-order interactions while maintaining computational feasibility.

Conclusion

This paper makes a substantial contribution to the field of machine learning interpretability by dissecting the problem of Most Influential Subset Selection. Through a combination of theoretical insights and empirical validation, it challenges the adequacy of existing methods and introduces a more adaptive approach that shows promise in addressing the complexities of collective data influence. Future research will likely continue to explore the trade-offs between computational cost and the accuracy of influential subset selection, possibly leading to even more efficient and robust methods. The paper sets a solid foundation for such advancements, thereby enhancing the transparency and trustworthiness of machine learning models.