Few-Shot Self-Rationalization with Natural Language Prompts (2111.08284v2)

Abstract: Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB -- a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.

Insights on Few-Shot Self-Rationalization with Natural Language Prompts

The paper "Few-Shot Self-Rationalization with Natural Language Prompts" by Ana Marasovi et al. investigates an innovative approach aimed at enhancing self-rationalization in NLP models using limited examples. Self-rationalization models are designed to generate task labels alongside free-text explanations, fostering model understandability which is crucial for user interactions. Typically, these models rely on extensive datasets of human-written explanations. However, this requirement limits the scalability of self-rationalization models to new tasks. The authors propose exploring few-shot self-rationalization as a pragmatic solution where models are prompted with minimal examples.

Methodology

The authors presented a standardized benchmark, termed the Few Explanations Benchmark (FEB), which consolidates four datasets providing English-language tasks with free-text explanations. These datasets span diverse tasks such as natural language inference and commonsense reasoning. The focus was on optimizing prompt design, employing different types of natural language prompts to guide the model's self-rationalization capability effectively.

Three types of prompt structures were evaluated:

1. QA Prompts: Utilizing UnifiedQA and T5, the paper examined multiple question-based formulations, finding that simple "What is...?" queries paired with task-specific tags facilitate accurate self-rationalization.
2. Infilling Prompts: These use preformatted gaps for generating explanations by prompting T5 to fill in template-based inputs. A more natural infilling prompt showed modest improvements over a basic version, although task-specific effectiveness varied.
3. Direct T5 Influence-Mimicking Prompts: By mimicking the training tasks of T5, these prompts aimed to leverage pre-trained strengths directly related to the tasks evaluated.

Key Findings

Experimentation revealed that scaling up model size generally enhanced performance significantly, with the largest models demonstrating improvement across various metrics: plausibility of explanations and accuracy in task prediction. Notably, the T5 models performed best with E-SNLI using infilling prompts, whereas UnifiedQA with QA prompts was superior for other datasets. The authors conducted human evaluations to validate the generated explanations, revealing that model size inversely correlated with the gap in performance relative to human-level outputs. However, models still lag behind human baselines in plausibility, emphasizing the scope for further refinement.

Implications and Future Work

This paper underscores few-shot self-rationalization as a feasible path forward for making NLP models more explainable, without extensive pre-existing labeled datasets. The FEB benchmark serves as a critical resource for evaluating progress in self-rationalization, encouraging research that can enhance model adaptability to new tasks with minimal data. While improvements in self-rationalization with larger models present an encouraging trend, the persistent gap relative to human-authored explanations indicates the necessity for further research.

Future developments may include:

• Exploring more sophisticated prompting methods, such as continuous prompt optimization and context-aware learning mechanisms.
• Integrating more complex reasoning capabilities that better mimic human explanatory patterns.
• Further advancing model compression and efficiency techniques to make larger models accessible in practical applications.

Overall, the paper establishes a robust framework for experimenting with few-shot learning paradigms in NLP, highlighting the necessity and possibility of self-rationalization that promises more intuitive and reliable NLP systems.