Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey (2201.12438v1)

Abstract: While commonsense knowledge acquisition and reasoning has traditionally been a core research topic in the knowledge representation and reasoning community, recent years have seen a surge of interest in the natural language processing community in developing pre-trained models and testing their ability to address a variety of newly designed commonsense knowledge reasoning and generation tasks. This paper presents a survey of these tasks, discusses the strengths and weaknesses of state-of-the-art pre-trained models for commonsense reasoning and generation as revealed by these tasks, and reflects on future research directions.

Commonsense Knowledge Reasoning and Generation with Pre-trained LLMs

The paper "Commonsense Knowledge Reasoning and Generation with Pre-trained LLMs: A Survey" by Prajjwal Bhargava and Vincent Ng provides a comprehensive examination of pre-trained LLMs (PLMs) and their application in commonsense reasoning and generation tasks. The authors delineate the landscape of how PLMs can be employed to navigate tasks traditionally seen as complex due to their reliance on commonsense knowledge.

Overview of Pre-trained LLMs

The emergence of PLMs has fundamentally altered the approach to NLP. The models rely on a self-supervised learning paradigm, enabling them to develop knowledge without requiring large labeled datasets. PLMs such as BERT, GPT, and T5 have demonstrated significant capabilities in encoding language representations that encompass both linguistic and commonsense nuances.

Capturing Commonsense Knowledge

A primary focus of the paper is assessing how well PLMs capture commonsense knowledge. It is evident from various probing studies that while PLMs show promise as alternatives to knowledge bases, they tend to struggle with generalized inference on unseen entities due to their propensity towards memorization during pre-training. Furthermore, PLMs can perform well in tasks requiring inference of physical properties or ontological knowledge but are less effective in learning widely accepted human properties from large corpora.

Commonsense Reasoning with PLMs

The paper scrutinizes the ability of PLMs to engage in commonsense reasoning across several axes:

1. Linguistic Reasoning: BERT is found lacking in sensitivity to linguistic nuances, particularly in negated sentences or those requiring complex logical reasoning.
2. Physical World Reasoning: PLMs can make inferences related to object affordances but struggle with unconventional usages. Integration with world dynamics can potentially augment their reasoning capabilities.
3. Abductive Reasoning: PLMs tend to falter when contexts require cross-sentence interpretation and complex temporal or causal inferences, demonstrating a gap between human and machine reasoning.
4. Social Reasoning: In scenarios involving social interactions, PLMs show varied performance, often better with emotion-centric questions but less consistent with spatial commonsense.
5. Multimodal Reasoning: Combining textual and visual modalities enhances reasoning performance, suggesting the utility of visual inputs in enriching LLM inferences.
6. Temporal Reasoning: PLMs face challenges in understanding temporal attributes and relations between events due to the scarcity of structured temporal knowledge bases.

Generating Commonsense Knowledge

When tasked with generating commonsense knowledge, PLMs exhibit limitations in coherency, concept coverage, and reasoning transparency. Efforts to improve these aspects involve adopting prototypes for sentence generation, leveraging knowledge graphs, and activating rich multi-hop reasoning over relational paths. Iterative refinement techniques also show promise in enhancing text generation quality.

Challenges and Future Directions

The authors identify several challenges and avenues for future research:

• Improving Benchmarks: Enhancing benchmarks to ensure they reflect true linguistic understanding and commonsense reasoning capabilities.
• Reducing Biases: Eliminating dataset biases that allow models to shortcut reasoning processes.
• Addressing Reporting Bias: Tackling the challenge of underreported knowledge in text corpora that leads to generalized inference errors.
• Enriching Knowledge Graphs: Developing strategies to densify and contextualize existing knowledge structures for enhanced commonsense reasoning.
• Exploring Multilinguality: Investigating how PLMs perform in multilingual settings, an area with significant potential for development.

In sum, the paper provides an exhaustive survey of the current state and future potential of PLMs in the context of commonsense knowledge reasoning and generation. It encourages continued exploration into the integration of various modalities, expanded knowledge resources, and optimized model architectures to advance the AI field further.