BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (1904.02232v2)

Abstract: Question-answering plays an important role in e-commerce as it allows potential customers to actively seek crucial information about products or services to help their purchase decision making. Inspired by the recent success of machine reading comprehension (MRC) on formal documents, this paper explores the potential of turning customer reviews into a large source of knowledge that can be exploited to answer user questions.~We call this problem Review Reading Comprehension (RRC). To the best of our knowledge, no existing work has been done on RRC. In this work, we first build an RRC dataset called ReviewRC based on a popular benchmark for aspect-based sentiment analysis. Since ReviewRC has limited training examples for RRC (and also for aspect-based sentiment analysis), we then explore a novel post-training approach on the popular LLM BERT to enhance the performance of fine-tuning of BERT for RRC. To show the generality of the approach, the proposed post-training is also applied to some other review-based tasks such as aspect extraction and aspect sentiment classification in aspect-based sentiment analysis. Experimental results demonstrate that the proposed post-training is highly effective. The datasets and code are available at https://www.cs.uic.edu/~hxu/.

• The paper introduces a post-training technique that adapts BERT for review reading comprehension and aspect sentiment tasks.
• The method leverages domain-specific review data and MRC datasets to boost performance in RRC, AE, and ASC tasks.
• Empirical results show significant F1 score improvements over baselines, validating the impact of specialized domain and task adaptation.

Overview of BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis

This paper presents a novel approach aimed at enhancing the utility of BERT in the context of Review Reading Comprehension (RRC) and Aspect-based Sentiment Analysis (ABSA). The research addresses the gap in leveraging product reviews for automated question-answering in e-commerce, introducing the RRC task which exploits user-generated reviews as knowledge sources. The authors propose a specialized post-training methodology to address the limitations present in domain-specific applications of BERT.

RRC and ABSA in E-Commerce

The paper emphasizes the role of question-answering systems in online commerce where existing methods, such as Community Question-Answering (CQA), often result in delays and unanswered queries. This necessitates an approach capable of deriving answers from customer reviews, which are a rich source of experiential insights. The newly introduced RRC task requires identifying precise text spans within reviews that answer user queries.

Methodology

The authors develop a dataset called ReviewRC and utilize BERT as the base model for their experiments, following a three-step training paradigm:

1. Domain Awareness Enhancement: Utilize a large corpus of domain-specific review data for masked LLM (MLM) and next sentence prediction (NSP) pre-training, injecting domain-focused knowledge into BERT.
2. Task Knowledge Integration: Use an existing large-scale MRC dataset, SQuAD, to incorporate task-specific comprehension abilities into BERT.
3. Fine-tuning: Adjust the BERT model's parameters using the annotated ReviewRC datasets tailored for RRC.

Experimental Results

The proposed post-training approach demonstrates significant performance improvements across RRC, AE, and ASC tasks when benchmarked against existing models like DrQA and even the vanilla BERT configuration. Specifically:

• RRC Effectiveness: BERT-PT outperforms both baseline and fine-tuning-only models, showcasing superior F1 scores attributable to enhanced extraction capabilities and domain adaptation.
• Aspect Extraction: Significant gains are achieved through domain-specific post-training, emphasizing that domain knowledge is crucial for identifying relevant aspects in reviews.
• Aspect Sentiment Classification: Incorporating task knowledge through MRC post-training boosts performance, underlining the benefits of task-oriented pre-training in sentiment analysis contexts.

Implications and Future Directions

This research advances RRC, establishing it as a viable method for extending machine comprehension capabilities to subjective and informal review data. The dual-stage post-training method could inform future development of domain and task-adaptive LLMs beyond e-commerce. There is potential for applying these findings in other domains requiring nuanced understanding of context and sentiment, such as social media analytics.

Moreover, future work may explore scaling the approach with larger BERT models or newer architectures like Transformers, potentially further enhancing performance. Additionally, more efficient pre-training techniques could be investigated to reduce computational demands without sacrificing accuracy and robustness.

In summary, this paper contributes a methodologically sound and empirically validated technique to harness BERT’s full potential for product-related natural language tasks, expanding the functional breadth of AI-driven e-commerce solutions.