CAVE: Controllable Authorship Verification Explanations (2406.16672v2)

Abstract: Authorship Verification (AV) (do two documents have the same author?) is essential in many sensitive real-life applications. AV is often used in proprietary domains that require a private, offline model, making SOTA online models like ChatGPT undesirable. Current offline models however have lower downstream utility due to low accuracy/scalability (eg: traditional stylometry AV systems) and lack of accessible post-hoc explanations. In this work, we take the first step to address the above challenges with our trained, offline Llama-3-8B model CAVE (Controllable Authorship Verification Explanations): CAVE generates free-text AV explanations that are controlled to be (1) structured (can be decomposed into sub-explanations in terms of relevant linguistic features), and (2) easily verified for explanation-label consistency (via intermediate labels in sub-explanations). We first engineer a prompt that can generate silver training data from a SOTA teacher model in the desired CAVE output format. We then filter and distill this data into a pretrained Llama-3-8B, our carefully selected student model. Results on three difficult AV datasets IMDb62, Blog-Auth, and Fanfiction show that CAVE generates high quality explanations (as measured by automatic and human evaluation) as well as competitive task accuracies.

• The paper introduces CAVE, which leverages LLMs to generate modular, free-text rationales that clearly align with intermediate linguistic features.
• The methodology distills GPT-4-Turbo outputs into a local LLaMa-3-8B model, achieving competitive accuracy and strong rationale consistency on challenging AV datasets.
• The approach improves data security and practical usability in sensitive fields like forensic analysis and plagiarism detection by providing interpretable, verifiable explanations.

Overview of "CAVE: Controllable Authorship Verification Explanations"

"Controllable Authorship Verification Explanations" (CAVE) is a research work aimed at enhancing the interpretability and security of Authorship Verification (AV) systems. Traditional AV tasks ascertain whether two documents share the same author by analyzing stylistic features or vector embeddings. However, these methods either lack scalability or interpretability, posing challenges in sensitive real-world applications such as forensic analysis, plagiarism detection, and misinformation analysis.

Methodology

The paper introduces "Cave," an AV model that provides structured, consistent explanations in natural text. Unlike traditional methods dependent on hand-crafted features or black-box neural architectures, Cave leverages LLMs to generate free-text rationales. These rationales are not only there to explain the AV decisions but are crafted to be modular and composed of intermediate labels for enhanced transparency.

Key Aspects of Cave:

1. Structured Rationales: Cave's rationales are designed to be decomposable into sub-explanations corresponding to distinct linguistic features. This structured approach makes the overall explanation more accessible and easier to verify.
2. Consistency: The rationales include intermediate labels for each feature, and the final AV decision is consistent with these intermediate steps.
3. Training Data: The model is distilled from a large LLM (GPT-4-Turbo) to a smaller, local LLM (LLaMa-3-8B). The training data consists of silver-standard rationales generated by GPT-4-Turbo and filtered according to specific metrics to ensure quality.

Experimental Evaluation

The authors tested Cave on three difficult AV datasets: IMDb62, Blog-Auth, and FanFiction. The results indicate that Cave achieves competitive task accuracies and high rationale qualities, as evidenced by both automatic and human evaluations.

Automatic Evaluation:

1. Accuracy: Cave demonstrated task accuracy competitive with existing state-of-the-art systems.
2. Consistency: The model showed high consistency between the rationales and the final labels, ensuring that the explanations can be trusted.

Human Evaluation:

A pilot paper involving human annotators assessed the quality of the rationales across several dimensions:

1. Detail-Consistency: Whether the rationale details were consistent with the input documents.
2. Factual-Correctness: Whether the rationales were factually accurate.
3. Label-Consistency: Whether each individual rationale segment was consistent with its intermediate label.

Implications

Practical Implications:

1. Security: By distilling a local model, Cave ensures that sensitive data does not have to be sent to online APIs, improving data security.
2. Usability: The structured format of the explanations makes them easier to parse and understand for end-users, such as legal professionals or forensic analysts, who require high levels of transparency.

Theoretical Implications:

1. Explainability in AV: This work advances the field by providing a practical approach to generating explanations that are both interpretable and consistent.
2. Balancing Accuracy and Explainability: The research underscores the importance of balancing these two metrics, showing that it is possible to achieve competitive accuracies while maintaining high-quality explanations.

Future Work

While Cave represents a significant advancement, future work could address:

1. Completeness of Rationales: The current model may miss some critical similarities or differences between documents. Future studies could develop metrics to ensure the completeness of explanations.
2. Error Analysis: Addressing systematic errors such as hallucinated details or dataset biases can further improve the model's reliability.
3. Dynamic Weighing: Implementing dynamic weighing of linguistic features at inference time could make the model more robust across diverse datasets.

Conclusion

Cave marks a notable step towards enhancing the transparency and interpretability of AV systems. By generating structured, consistent explanations, it bridges the gap between the need for scalable, accurate models and the requirement for human-understandable, trustworthy outputs. Through this work, the authors contribute significantly to both the theory and practice of AV, setting the stage for further advancements in explainable AI.