Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach (2408.07238v1)

Abstract: Advanced LLMs like GPT-4 or LlaMa 3 provide superior performance in complex human-like interactions. But they are costly, or too large for edge devices such as smartphones and harder to self-host, leading to security and privacy concerns. This paper introduces a novel interpretable knowledge distillation approach to enhance the performance of smaller, more economical LLMs that firms can self-host. We study this problem in the context of building a customer service agent aimed at achieving high customer satisfaction through goal-oriented dialogues. Unlike traditional knowledge distillation, where the "student" model learns directly from the "teacher" model's responses via fine-tuning, our interpretable "strategy" teaching approach involves the teacher providing strategies to improve the student's performance in various scenarios. This method alternates between a "scenario generation" step and a "strategies for improvement" step, creating a customized library of scenarios and optimized strategies for automated prompting. The method requires only black-box access to both student and teacher models; hence it can be used without manipulating model parameters. In our customer service application, the method improves performance, and the learned strategies are transferable to other LLMs and scenarios beyond the training set. The method's interpretabilty helps safeguard against potential harms through human audit.

Interpretable Knowledge Distillation for Enhancing Smaller LLMs

Introduction

The paper "Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach" by Tong Wang, K. Sudhir, and Dat Hong presents a novel knowledge distillation technique aimed at enhancing the performance of smaller, more economical LLMs by leveraging the knowledge of more advanced LLMs such as GPT-4 or LlaMa 3. This research is motivated by the cost, computational, and privacy limitations associated with deploying large LLMs, making smaller models an attractive option for businesses, particularly in customer service applications.

Methodology

The proposed approach centers on an interpretable strategy teaching mechanism, divergent from traditional knowledge distillation methods that typically rely on fine-tuning the student model directly based on the teacher model's outputs. Instead, this method involves generating a library of scenarios along with corresponding strategies. The library functions as an external knowledge base that the student LLM can query during deployment, thus enhancing the interpretability and safety of the distilled knowledge.

Scenario Generation

The process involves two LLMs interacting within a simulated environment: the teacher LLM and a customer LLM (simulated using GPT-4). The scenario generation phase alternates between using the teacher and student models to interact with the customer LLM. This alternating process aims to mitigate distribution shift, a common issue where student models fail to perform as expected in real-world settings because their training data was generated exclusively by the teacher model.

Strategy Teaching

The strategy teaching method involves an iterative refinement process where the teacher provides the student with specific, context-dependent strategies for handling various scenarios. Each iteration includes evaluating the student's performance and updating the strategies based on discrepancies between the student’s and teacher's responses. This iterative process continues until no further improvements are observed or a maximum number of iterations is reached.

Evaluation and Results

The effectiveness of the proposed method is validated using four different student models: LlaMa-2 7B, LlaMa-2 13B, LlaMa-2 70B, and GPT-3.5. Performance is evaluated in a customer service context, specifically in handling goal-oriented dialogue tasks aimed at achieving high customer satisfaction.

Main Findings

1. Effectiveness of Strategy Teaching: The strategy teaching approach consistently outperforms traditional fine-tuning methods. Fine-tuning focuses on lexical similarities and parameter adjustments, while strategy teaching imparts a deeper understanding of how to handle varying scenarios at a strategic level.
2. Scenario-Specific Strategies: Customized, context-specific strategies were shown to be more effective than global guidelines, which lack the adaptability needed for different scenarios.
3. Distribution Shift: Allowing the student to participate in scenario generation significantly reduces distribution shift, enhancing the relevance and applicability of the strategies during deployment.
4. Model and Context Transferability: The strategy library exhibits a degree of transferability across different student LLMs and customer service contexts, indicating the robustness and broad applicability of the distilled strategies. Larger LLMs like GPT-3.5 benefit more from this approach due to their superior capability in extrapolating and adapting strategies.

Human and LLM-Based Evaluation

The evaluation involves both human raters and advanced LLMs (GPT-4 and LlaMa 3 70B). Results indicate substantial improvements in customer satisfaction ratings for the student models trained using the proposed method. Human evaluations further corroborate these findings, showing that the student models are rated closer to the teacher’s performance.

Implications and Future Research

The paper demonstrates that smaller LLMs can achieve near-parity with more advanced models in specific tasks through an interpretable knowledge distillation approach. This has significant practical implications, as it reduces the dependency on high-cost, high-complexity models, making advanced AI solutions more accessible. By storing strategies in an interpretable library, the approach ensures transparency and facilitates human oversight, thereby enhancing AI safety and trustworthiness.

Future research could explore extending this method to other domains and tasks, experimenting with different student-teacher model pairs, and further refining the retrieval mechanisms for scenario-specific strategies. Additionally, there is potential in investigating the impact of various types of customer interactions on the effectiveness of the distilled strategies.

Conclusion

The interpretable knowledge distillation approach presented in this paper offers a compelling method for enhancing smaller LLMs by leveraging the strategic insights of more advanced LLMs. By focusing on strategy teaching, this method provides a robust, adaptable, and interpretable solution that addresses the inherent limitations of traditional knowledge distillation techniques. The results affirm that smaller LLMs, when guided by strategically distilled knowledge, can perform competitively in goal-oriented dialogue tasks, thereby broadening the applicability and feasibility of deploying LLMs in cost-sensitive and privacy-conscious environments.