Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering (2404.17723v2)

Abstract: In customer service technical support, swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries. The conventional retrieval methods in retrieval-augmented generation (RAG) for LLMs treat a large corpus of past issue tracking tickets as plain text, ignoring the crucial intra-issue structure and inter-issue relations, which limits performance. We introduce a novel customer service question-answering method that amalgamates RAG with a knowledge graph (KG). Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations. During the question-answering phase, our method parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration of a KG not only improves retrieval accuracy by preserving customer service structure information but also enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on our benchmark datasets, utilizing key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR) metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. Our method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%.

• The paper presents an innovative integration of knowledge graphs with retrieval-augmented generation to preserve the structural context of customer support tickets.
• It employs tree-based graph construction to capture complex relationships, resulting in a 77.6% improvement in mean reciprocal rank.
• The enhanced system demonstrated practical impact by reducing issue resolution time by 28.6% in real-world customer service settings.

Enhancing Customer Service Efficiency with Knowledge Graph-Augmented Question Answering Systems

Introduction to the Issue and Innovation

In the field of customer service, particularly when dealing with technical issues, efficiency and accuracy in providing answers are key factors that determine customer satisfaction. Traditional text-based systems for querying historical support tickets often face challenges related to the loss of structural information and connection between the tickets. A promising solution explored in this paper involves the integration of Knowledge Graphs (KG) with Retrieval-Augmented Generation (RAG) techniques aimed at refining the retrieval process and improving response quality in real customer service environments.

Key Challenges and Novel Approaches

Two main hurdles in traditional systems include:

• Compromised Retrieval: Structures such as issue connections and interactions are lost when tickets are simply broken down into text chunks.
• Reduced Answer Quality Due to Segmentation: Important parts of responses can be lost when tickets are indiscriminately split into parts to fit model constraints.

The paper presents an innovative approach that includes:

• Parsing issues into structured trees forming a comprehensive graph that maintains the intrinsic relationships among entities.
• Using these graphs during the retrieval process ensuring that structure and connection between issues are leveraged to boost the accuracy and contextuality of responses.

Integration of Knowledge Graph in RAG

1. Knowledge Graph Construction:
   • Historical issue tickets are parsed into a tree-based graph structure where nodes represent different sections of the issue (like summary, description, priority).
   • Each issue's tree is linked with others to form a broad graph, preserving and utilizing relationships, whether explicit (like ticket references) or implicit (semantic similarities).
2. Utilizing the Knowledge Graph for Query Answering:
   • User queries undergo entity extraction and intent detection.
   • The graph structure helps pinpoint relevant issues and their connections, focusing the retrieval to be more context-aware and accurate.

Empirical Validation and Results

The system's effectiveness was benchmarked against standard text-based retrieval systems through several well-established metrics:

• Retrieval Metrics: The system showed a significant 77.6% improvement in Mean Reciprocal Rank (MRR), indicating better initial retrieval.
• Question Answering Metrics: Enhancements in BLEU, ROUGE, and METEOR scores demonstrated that answers were not only more relevant but also contextually richer.

Real-world application within LinkedIn’s customer service team showed a notable reduction in issue resolution time by 28.6%, substantiating the practical value of this approach.

Implications and Future Directions

This paper reveals substantial improvements in automated customer service systems through the integration of knowledge graphs, suggesting potential for broader applications beyond the current use-case. Future research could explore:

• Automated updates to the knowledge graph in response to new queries or ticket information.
• Expanding to other domains or types of customer service environments, like healthcare or public service.
• Additional refinement of graph structure extraction and parsing algorithms for enhanced adaptability and response accuracy.

Concluding Thoughts

The integration of KG into RAG frameworks marks a significant advancement in leveraging historical data for customer service. By maintaining the inherent structure of past tickets and intelligently utilizing this structured information, the method detailed in this paper not only enhances retrieval accuracy but also dramatically improves the quality of generated responses, paving the path for smarter, more responsive customer service systems. Future explorations in this direction seem not only promising but necessary, as businesses continue to seek ways to enhance efficiency in customer interaction and problem resolution.