Understanding Pre-trained BERT for Aspect-based Sentiment Analysis (2011.00169v1)

Abstract: This paper analyzes the pre-trained hidden representations learned from reviews on BERT for tasks in aspect-based sentiment analysis (ABSA). Our work is motivated by the recent progress in BERT-based LLMs for ABSA. However, it is not clear how the general proxy task of (masked) LLM trained on unlabeled corpus without annotations of aspects or opinions can provide important features for downstream tasks in ABSA. By leveraging the annotated datasets in ABSA, we investigate both the attentions and the learned representations of BERT pre-trained on reviews. We found that BERT uses very few self-attention heads to encode context words (such as prepositions or pronouns that indicating an aspect) and opinion words for an aspect. Most features in the representation of an aspect are dedicated to the fine-grained semantics of the domain (or product category) and the aspect itself, instead of carrying summarized opinions from its context. We hope this investigation can help future research in improving self-supervised learning, unsupervised learning and fine-tuning for ABSA. The pre-trained model and code can be found at https://github.com/howardhsu/BERT-for-RRC-ABSA.

Understanding Pre-trained BERT for Aspect-based Sentiment Analysis

The paper "Understanding Pre-trained BERT for Aspect-based Sentiment Analysis" by Hu Xu and colleagues presents an empirical investigation into the functionality of BERT's pre-trained hidden representations in the context of Aspect-Based Sentiment Analysis (ABSA). This research specifically aims to discern how the general pretext task of masked LLMing (MLM) in BERT can contribute to the specialized demands of ABSA.

Key Insights

The paper explores BERT's self-attention mechanisms and the learned representations when applied to ABSA tasks, particularly focusing on aspect extraction (AE) and aspect sentiment classification (ASC). The authors provide a nuanced analysis of BERT's performance in encoding contextual information, domain semantics, and the limitations of its sentiment capture capabilities.

1. Attention Heads in BERT: The analysis reveals that BERT utilizes only a few self-attention heads to capture important context and opinion words related to aspects. The implication here is that the majority of attention is allocated towards encoding domain-specific semantics rather than opinion summarization.
2. Hidden Representations: Through dimensionality reduction techniques, the paper demonstrates that BERT's representation space is heavily influenced by domain knowledge, with domain separation well-pronounced. However, aspect sentiment does not distinctly shape the representation space, suggesting a deficiency in carrying polarity information.
3. Impact of Masked LLMing: The findings suggest that while BERT shows effectiveness in extracting aspects, the MLM task is insufficient for capturing opinion features. The representation of aspect words, primarily fine-tuned for semantics, often fails to encapsulate sentiment, indicating a potential limitation in applying BERT directly for ABSA without further task-specific adjustments.

Implications and Future Directions

The work importantly emphasizes that current pre-training strategies in BERT, such as MLM, may not fully align with the granular requirements of ABSA. The research suggests the necessity for developing novel self-supervised tasks that better disentangle aspect and sentiment features. Possible directions include leveraging weak supervisory signals, such as item-group reviews or ratings, to enhance BERT’s feature learning in sentiment dimensions.

Future research should explore alternative pre-training paradigms that can inherently distinguish aspect features while capturing sentiment nuances. Expanding the latent space allocation for sentiment-specific features or integrating domain adaptation strategies could enhance the efficacy of LMs in ABSA.

The paper's findings are crucial for advancing the development of more robust LM architectures that are better suited for complex sentiment analysis tasks, indicating a pressing need for continued exploration in pre-training task design and representation learning.