Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification (1908.11860v2)

Abstract: Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce, where data and insights from reviews can be leveraged to create value for businesses and customers. Recently, deep transfer-learning methods have been applied successfully to a myriad of NLP tasks, including ATSC. Building on top of the prominent BERT LLM, we approach ATSC using a two-step procedure: self-supervised domain-specific BERT LLM finetuning, followed by supervised task-specific finetuning. Our findings on how to best exploit domain-specific LLM finetuning enable us to produce new state-of-the-art performance on the SemEval 2014 Task 4 restaurants dataset. In addition, to explore the real-world robustness of our models, we perform cross-domain evaluation. We show that a cross-domain adapted BERT LLM performs significantly better than strong baseline models like vanilla BERT-base and XLNet-base. Finally, we conduct a case study to interpret model prediction errors.

Aspect-Target Sentiment Classification via Domain Adaptation with BERT

This paper explores Aspect-Target Sentiment Classification (ATSC), a refined domain of Aspect-Based Sentiment Analysis (ABSA), utilizing a method that involves fine-tuning the BERT LLM. The core objective is to enhance sentiment classification performance by applying domain adaptation techniques on BERT, which has been pre-trained on a large general corpus. The authors propose a novel two-step procedure: perform domain-specific fine-tuning of BERT in a self-supervised manner, followed by task-specific supervised fine-tuning for ATSC. This methodological enhancement yielded new state-of-the-art results on the SemEval 2014 Task 4 dataset for the restaurants domain.

The research emphasizes the importance of domain-specific knowledge in enhancing the performance of pre-trained LLMs when applied to sentiment analysis tasks. By adapting BERT to specific domains (e.g., restaurants and laptops) through fine-tuning, the authors achieved superior performance over the traditional vanilla BERT-base and XLNet-base models.

Key Methodology

The paper introduces a systematic approach to tackle ATSC:

1. Self-supervised Domain-specific Fine-tuning: BERT is first fine-tuned using large-scale domain-specific corpora (Yelp reviews for restaurants and Amazon reviews for laptops). This step is crucial to imbue BERT with domain-relevant linguistic subtleties not captured by pre-training on general corpora like Wikipedia.
2. Supervised Task-specific Fine-tuning: Following domain adaptation, the fine-tuned model further undergoes training on the ATSC task using the target domain-specific dataset. This step ensures that the model learns to associate sentiment polarity with specific aspect-target pairs present in the sentences.

Numerical Findings and Insights

Upon implementing this two-step fine-tuning approach, the research demonstrates significant improvements in ATSC performance metrics, with notable numerical advances:

• Achieving new state-of-the-art accuracy on the SemEval 2014 Task 4 restaurants dataset, indicating the effectiveness of domain-specific LLM adaptation.
• Cross-domain evaluation shows adapted models offer strong performance superiority over unadapted ones, enhancing the robustness and real-world applicability of the models.

The paper particularly highlights the importance of the number of domain-specific fine-tuning iterations. Initially, performance increases rapidly; however, improvements plateau after a threshold, signifying the diminishing returns of continuous fine-tuning.

Implications and Future Directions

The implications of this research are multifaceted. Practically, it demonstrates that even state-of-the-art LLMs like BERT can be substantially improved through adaptation to domain-specific datasets, a salient takeaway for applying NLP models in industry settings where domain-relevance is critical—such as e-commerce and service reviews.

Theoretically, this paper contributes to a deeper understanding of the interaction between pre-trained models and domain-specific data. It opens pathways for further exploration in cross-domain adaptations, suggesting that fine-tuning strategies could generalize across more varied domains including those less similar like hotels or services.

Future research might investigate fine-tuning strategies leveraging XLNet, which has shown competitive performance. Additionally, extending cross-domain evaluations could deepen the understanding of sentiment transferability and guide custom adaptations for diverse real-world applications.

In conclusion, this paper showcases the potential of domain-adapted BERT models in advancing aspect-target sentiment classification, underscoring the continued evolution of LLMing techniques in specialized NLP applications.