CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs (2404.01343v4)

Abstract: Businesses and software platforms are increasingly turning to LLMs such as GPT-3.5, GPT-4, GLM-3, and LLaMa-2 for chat assistance with file access or as reasoning agents for customer service. However, current LLM-based customer service models have limited integration with customer profiles and lack the operational capabilities necessary for effective service. Moreover, existing API integrations emphasize diversity over the precision and error avoidance essential in real-world customer service scenarios. To address these issues, we propose an LLM agent named CHOPS (CHat with custOmer Profile in existing System), designed to: (1) efficiently utilize existing databases or systems for accessing user information or interacting with these systems following existing guidelines; (2) provide accurate and reasonable responses or carry out required operations in the system while avoiding harmful operations; and (3) leverage a combination of small and large LLMs to achieve satisfying performance at a reasonable inference cost. We introduce a practical dataset, the CPHOS-dataset, which includes a database, guiding files, and QA pairs collected from CPHOS, an online platform that facilitates the organization of simulated Physics Olympiads for high school teachers and students. We have conducted extensive experiments to validate the performance of our proposed CHOPS architecture using the CPHOS-dataset, with the aim of demonstrating how LLMs can enhance or serve as alternatives to human customer service. Code for our proposed architecture and dataset can be found at {https://github.com/JingzheShi/CHOPS}.

• The paper presents the CHOPS framework that integrates LLMs with customer profiles to achieve context-aware customer service.
• It introduces a Classifier-Executor-Verifier architecture that optimizes query processing and ensures response accuracy.
• Experimental results on the CPHOS-dataset demonstrate improved cost-effectiveness and reliability in sensitive customer service settings.

The paper "CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs" introduces a novel framework leveraging LLMs to enhance customer service automation capabilities by effectively integrating these models with customer profile data and existing operational systems. The authors propose a system named CHOPS, which stands for Chat with custOmer Profile in existing System. This system is designed to address limitations in current LLM applications that do not sufficiently incorporate customer profiles or operational integration capabilities necessary for precise and error-free interactions in customer service contexts.

Key Contributions:

1. Integration with Customer Profiles: CHOPS was developed to efficiently utilize existing databases and systems, ensuring it can access user information or interact with these systems based on predefined guidelines. By doing so, it aims to provide accurate and contextually relevant responses while minimizing the risk of executing harmful or incorrect operations.
2. Architecture Design:
   • The CHOPS framework employs a Classifier-Executor-Verifier architecture to enhance interaction accuracy and control inference costs.
   • Classifier: Determines whether a user query relates to guide files, API usage, or requires basic user information.
   • Executor: Processes the user query using the extracted information to generate responses or execute operations.
   • Verifier: Checks the executor's responses to ensure they are correct before finalizing the interaction with users, a structure inspired by previous advancements in self-verification methods to improve model reliability.
3. Focus on API Integration: Unlike previous approaches focused on widely diverse API sets, CHOPS narrows down the utilization to critical APIs, thereby prioritizing high accuracy in user status modifications, particularly in sensitive sectors like banking. This approach enlightens the need for specific customer service datasets that reflect real-world scenarios more accurately.
4. CPHOS-dataset: To facilitate thorough evaluation, the researchers introduce a novel dataset named CPHOS, derived from an online platform used to simulate Physics Olympiads. The dataset includes databases, guiding files, and collected question-answer pairs. This practical dataset aims to bridge existing gaps by simulating real interactions requiring database queries or the utilization of guiding files.

Experimental Validation:

The authors report extensive experimentation demonstrating the efficacy of the CHOPS architecture. Testing with the CPHOS-dataset, they claim that their architecture can achieve substantial improvements in customer service accuracy by outperforming a naive use of more powerful LLMs in terms of both cost-effectiveness and reliability. Specifically, performance improvements were consistent even when leveraging smaller model variants due to the architectural specialization of CHOPS.

Technical Details:

• Use of LLMs: LLMs like GPT-3.5, GPT-4, GLM-3, and LLaMa-2 are exploited for their ability to generate natural language responses. The integration emphasizes blending model strengths across tasks, balancing performance and inference costs.
• RAG Framework: Retrieval-Augmented Generation (RAG) techniques are employed to incorporate external knowledge sources into LLM responses efficiently, aligning with vector-based data retrieval systems.

Conclusion:

The framework proposes a refined method to integrate LLMs into practical customer service systems effectively. The results suggest that the CHOPS architecture not only enhances interaction accuracy and cost efficiency but also establishes a flexible design paradigm proficient at accommodating LLMs within operational settings that require interaction with both data-driven APIs and directive guide documents. The development and usage of the CPHOS-dataset further illustrate the framework’s applicability in practical business scenarios, laying a prospective pathway for future customer service automation tools that integrate advanced LLMs.