RISE: Randomized Input Sampling for Explanation of Black-box Models (1806.07421v3)

Abstract: Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/

• The paper introduces RISE, a novel black-box approach that generates pixel importance maps for deep neural networks using randomized binary masks.
• It employs innovative deletion and insertion metrics to objectively evaluate the impact of pixel regions on image classification outcomes.
• The method demonstrates broad applicability and improved performance over white-box techniques, supporting transparency in high-stakes applications.

Analyzing the RISE Approach for Explainable AI

The paper "RISE: Randomized Input Sampling for Explanation of Black-box Models" by Vitali Petsiuk, Abir Das, and Kate Saenko introduces a novel approach for generating explanations for black-box models, specifically deep neural networks for image classification. The method, termed RISE, generates an importance map that elucidates the significance of each pixel in contributing to the model's decision.

Methodology

Fundamentally, RISE operates on black-box models, which means the method does not rely on the internal workings of the model such as gradient information, intermediate activations, or weights. This is a significant advantage because it allows RISE to be universally applicable to any model without requiring modifications or access to the model's internals.

RISE generates importance maps by sampling random binary masks to obscure parts of the input image and then passing these masked images through the model to observe the output class probabilities. Specifically:

• For each pixel, the importance is empirically estimated by running the model on multiple masked versions of the input image.
• The final importance map is a weighted combination of the random masks, where the weights correspond to the model's output probabilities.

This approach contrasts with existing white-box methods like Grad-CAM, which utilizes gradient information, and LIME, which creates linear surrogate models around each prediction. While methods like Grad-CAM are restricted to certain network architectures and require full access to the model's internals, RISE maintains its black-box nature, providing a more generalized solution.

Evaluation Metrics

To evaluate RISE, two novel automatic metrics are proposed:

1. Deletion Score: This metric assesses how the model's output probability for a class changes as important pixels (as indicated by the importance map) are progressively removed from the image. A rapid drop in class probability results in a lower deletion score, indicating a more accurate saliency map.
2. Insertion Score: This metric evaluates how the model's class probability increases as pixels are progressively added to an otherwise blurred image, following the order of importance. Higher insertion scores indicate better explanations as critical pixels are identified quickly.

Both these metrics aim to offer an objective, causal evaluation that captures the actual importance of pixels in the model's decision-making process, free from human bias inherent in traditional metrics such as the pointing game.

Empirical Results

The RISE method was subjected to extensive experiments across several datasets, including ImageNet, PASCAL VOC, and MSCOCO. Comparative studies with existing techniques such as Grad-CAM and LIME yield the following insights:

• Deletion and Insertion Scores: RISE outperforms other methods, including Grad-CAM, in terms of both deletion and insertion scores, with significant improvements noted particularly with the ResNet50 and VGG16 network architectures. This effectively demonstrates the suitability of RISE in identifying the most critical features influencing the model's predictions.
• Pointing Game Accuracy: While the pointing game metric (which evaluates how well the highest importance points overlap with human-annotated object regions) is more human-centric, RISE still performs competitively. Particularly, for the VGG16 network on PASCAL VOC dataset, RISE achieves higher accuracy than several state-of-the-art white-box methods.

Implications and Future Research

The significance of RISE lies in its black-box nature, offering a robust tool for explainability without necessitating access to the intricacies of the model. This broad applicability is crucial for the deployment of explainable AI (XAI) in real-world scenarios where model internals may not be accessible due to proprietary or technical constraints.

From a practical perspective, RISE can be invaluable in domains requiring high transparency, such as medical diagnostics, autonomous vehicles, and legal systems. The objective nature of its evaluation metrics further supports its adoption, providing clearer insights into the model's decision-making process than metrics tied to human annotations.

Future research could explore:

• Optimization of Mask Sampling: Enhancing the efficiency of the method by reducing the number of required mask samples without compromising the accuracy of the importance map.
• Expanding to Other Domains: Adapting RISE for other complex models including those used in video analysis or sequential data, expanding its utility beyond image classification.
• Integration with Human-Centric Methods: Combining the causal metrics proposed by RISE with human-centric evaluation techniques for a more holistic approach to model explainability.

In conclusion, RISE presents a significant advancement in explainable AI by offering a universal, black-box method for generating accurate importance maps. Its rigorous evaluation through novel causal metrics underscores its efficacy and potential for broad application across complex neural network models.