Evaluating and Characterizing Human Rationales (2010.04736v1)

Abstract: Two main approaches for evaluating the quality of machine-generated rationales are: 1) using human rationales as a gold standard; and 2) automated metrics based on how rationales affect model behavior. An open question, however, is how human rationales fare with these automatic metrics. Analyzing a variety of datasets and models, we find that human rationales do not necessarily perform well on these metrics. To unpack this finding, we propose improved metrics to account for model-dependent baseline performance. We then propose two methods to further characterize rationale quality, one based on model retraining and one on using "fidelity curves" to reveal properties such as irrelevance and redundancy. Our work leads to actionable suggestions for evaluating and characterizing rationales.

Insights into Evaluating and Characterizing Human Rationales in NLP

The paper "Evaluating and Characterizing Human Rationales" addresses a critical aspect of explainable AI, focusing on the evaluation of human-generated rationales versus machine-generated explanatory rationales. This paper emphasizes the need to scrutinize human rationales using automatic metrics and improve these metrics to enhance understanding.

Core Contributions

The research introduces a detailed analysis of the performance of human rationales across various datasets and models. The authors identify two prevailing strategies: using human-generated rationales as a gold standard and the assessment of rationales using automatic metrics based on model behavior, particularly sufficiency and comprehensiveness. The surprising insight from the paper is that human rationales do not inherently perform well on these automatic metrics, highlighting a potential discrepancy in what is deemed a "good" rationale.

To address this, the paper proposes:

1. Improved Metrics: The authors suggest normalization procedures to accommodate variations in model-dependent baseline performances, allowing for a fair comparison across different models.
2. Characterization Methods: They introduce methods centered on model retraining and fidelity curves to uncover properties like irrelevance and redundancy in rationales, thereby providing a more nuanced understanding of rationale quality.

Key Findings

• Model Dependence: Human rationales often demonstrate varying levels of sufficiency and comprehensiveness, significantly influenced by the models employed. For instance, the RoBERTa model, while highly accurate, demonstrated lower sufficiency scores for human rationales compared to simpler models, suggesting an inverse correlation between model accuracy and explanation sufficiency.
• Class Discrepancies: The comprehensiveness of rationales frequently differs among classes within the same dataset. This is particularly evident in tasks like WikiAttack, where rationales for absence-based classes like "no-attack" are inherently less comprehensive.

The paper also examines the implications of automatic metrics by introducing normalization procedures. These metrics adjust for inherent model biases, thereby offering a more precise evaluation of rationale fidelity.

Practical Implications

The research has several practical implications. The normalization of fidelity metrics ensures more reliable interpretation of model behavior, crucial for machine learning applications requiring explainability. Additionally, the introduction of fidelity curves provides insights into the intrinsic qualities of rationales, such as irrelevance and redundancy, which can be pivotal in refining datasets and designing more robust models.

Theoretical Implications and Future Directions

The authors suggest that these findings necessitate a reevaluation of the use of human rationales as a definitive gold standard in machine learning. The discrepancy between human rationales and automatic metrics points to a possible gap in understanding model-specific needs in rationale evaluation. Future research could explore the alignment of human rationales with model-specific decision-making processes, potentially leading to the development of more sophisticated explanation frameworks.

Potential future developments in AI include leveraging these insights to create hybrid models that better incorporate human rationales while maintaining high fidelity in automatic metrics. Additionally, such work could enhance methods for training models via explanations, fostering models that not only perform well but are also interpretable and aligned with human logic.

In conclusion, this paper presents a compelling discourse on the validity of human rationales in NLP and proposes actionable paths for improving rationale evaluation and characterization, contributing significantly to the field of explainable AI.