FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging (2012.15781v2)

Abstract: Influence functions approximate the "influences" of training data-points for test predictions and have a wide variety of applications. Despite the popularity, their computational cost does not scale well with model and training data size. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. We use k-Nearest Neighbors (kNN) to narrow the search space down to a subset of good candidate data points, identify the configurations that best balance the speed-quality trade-off in estimating the inverse Hessian-vector product, and introduce a fast parallel variant. Our proposed method achieves about 80X speedup while being highly correlated with the original influence values. With the availability of the fast influence functions, we demonstrate their usefulness in four applications. First, we examine whether influential data-points can "explain" test time behavior using the framework of simulatability. Second, we visualize the influence interactions between training and test data-points. Third, we show that we can correct model errors by additional fine-tuning on certain influential data-points, improving the accuracy of a trained MultiNLI model by 2.5% on the HANS dataset. Finally, we experiment with a similar setup but fine-tuning on datapoints not seen during training, improving the model accuracy by 2.8% and 1.7% on HANS and ANLI datasets respectively. Overall, our fast influence functions can be efficiently applied to large models and datasets, and our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors. Code is available at https://github.com/salesforce/fast-influence-functions

• The paper introduces FastIF, reducing influence computation time by 80-fold through kNN search, optimized Hessian inversion, and parallel processing.
• It employs kNN to target influential training points, effectively minimizing the candidate set and overall computational overhead.
• Experimental results demonstrate enhanced model debugging and error correction, with improvements such as a 2.8% accuracy gain on selected tasks.

FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

The paper "FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging" introduces innovative advancements in the domain of influence functions within machine learning models, specifically targeting the scalability and computational efficiency of these functions. The work builds upon the established foundation where influence functions are utilized to approximate the impact of individual training data points on a model's predictions at test time. These functions have applications in model debugging and interpretation, yet their computational demands have historically limited their practical utility, especially in the context of large models and datasets.

Key Contributions

The authors present FastIF, a methodology designed to significantly reduce the computational overhead associated with influence functions. The approach involves three primary enhancements:

1. $k$-Nearest Neighbors ($k$NN) Search: By utilizing $k$NN, the authors efficiently narrow down the training data candidate set to those more likely to be influential. This reduces the search space and thus the computation time required for influence estimation.
2. Optimized Inverse Hessian Estimation: The authors enhance the efficiency of estimating the inverse Hessian-vector product, a critical aspect of influence computation, by identifying configurations that strike a balance between speed and quality. This involves using small batch sizes and parallelizing the approximation process across multiple GPUs, yielding substantial speed-ups without significant loss of estimation quality.
3. Parallel Computing: By introducing a parallel variant of the influence function computation, the framework takes advantage of multi-GPU setups to further accelerate the process.

The FastIF methodology achieves approximately an 80-fold speedup in influence computation while maintaining a high correlation with the original influence values, demonstrating robust performance improvements without compromising accuracy.

Experimental Applications

FastIF's applications are demonstrated through a series of experiments that highlight its utility in both theoretical and practical contexts:

1. Model Behavior Explanation (Simulatability Framework): The authors test whether influential data points can enhance a simulator model's prediction of a task model's behavior, utilizing frameworks for model simulatability. Results show improved predictive accuracy when the simulator model is fine-tuned on identified influential examples, demonstrating FastIF's potential in model interpretability.
2. Visualization of Influence Interaction: By visualizing how training data points influence test data, the paper uncovers latent structures in datasets. This visual analysis provides insights into dataset characteristics, offering a novel approach for exploratory data analysis.
3. Error Correction and Model Improvement: FastIF allows for correcting model errors through fine-tuning on selected influential data points. Experiments with the MultiNLI model demonstrate substantial improvements on evaluation datasets (e.g., a 2.5% accuracy increase on the HANS dataset). Additionally, the methodology shows promise in leveraging unseen datasets for further performance enhancement through data augmentation (e.g., improving accuracy by over 2.8% on the ANLI dataset).

Implications and Future Directions

The research presented in this paper has several significant implications:

• Practical Scalability: By reducing the computational cost associated with influence functions, FastIF makes these diagnostics feasible for use with large-scale datasets and models, broadening the applicability of influence functions in real-world scenarios.
• Enhanced Model Diagnosis and Debugging: Researchers and practitioners can efficiently identify and mitigate problematic influences in their models, improving robustness and performance.
• Future AI Departmental Integration: The advancements brought forth by FastIF suggest potential integration into larger AI frameworks where model transparency and accountability are critical.

In conclusion, FastIF represents a valuable progression in making influence functions more accessible and practical for machine learning researchers. It bridges the gap between theoretical potential and practical application, underscoring areas for further research, such as extending the techniques to different model architectures and exploring other potential applications beyond interpretation and debugging.