- The paper demonstrates that humans recontextualize implausible SVO events into plausible figurative interpretations, unlike LLMs that display shallow, biased patterns.
- The experimental design using the PAP dataset reveals significant quantitative differences via chi-square, Cramér’s V, and Kendall’s τ, highlighting the gap in human versus LLM performance.
- Findings underscore the need for improved LLM architectures and refined prompt engineering to capture nuanced human semantic repair in figurative language.
Summary of "Contextualising (Im)plausible Events Triggers Figurative Language" (2604.07885)
Research Problem and Motivation
The paper investigates the interplay between plausibility and (non-)literalness in the interpretation of subject-verb-object (SVO) event triples in English. By interrogating both abstract and concrete constituent categories, the authors systematically examine how humans and LLMs contextualize and categorize plausible, implausible, literal, and figurative events. Prior work has focused on literal plausibility using neural embeddings and transformer-based models, but this study uniquely explores the intersection with figurative language, particularly exposing gaps in LLM contextualization strategies when compared to nuanced human judgments.
Experimental Design and Dataset
The study leverages a subset of the PAP dataset [Eichel and Schulte im Walde, 2023], consisting of 411 SVO triples annotated for plausibility and abstractness. Events are categorized on a scale from abstract (a), mid-range (m), to concrete (c) based on the concreteness ratings. The experimental protocol involves humans and four LLMs (Qwen3-4B, Gemma3-4B, Mistral-7B, Llama3.1-8B) making binary judgments on the literalness and providing sample contextualizing sentences. The collection yields 6,497 human and 14,555 LLM judgments, as well as 6,497 human and 3,288 LLM-generated sentences.
Major Findings
Human vs. LLM Judgments
- Nuanced Human Interpretation: Humans demonstrate robust capacity for making plausible interpretations even for initially nonsensical or whimsical SVO events, often recontextualizing into figurative readings when literal plausibility is absent.
- LLM Shallow Patterns: LLMs display a tendency toward interpreting implausible events as non-literal (figurative) but plausible, evidencing shallow context generation, and frequently failing to identify truly nonsensical or irreparable events. Notably, models produced overwhelmingly plausible and figurative readings, particularly under zero-shot prompts, with Llama3.1-8B more closely mirroring (but not matching) human literal interpretations.
- Prompting Effects: Few-shot prompts, especially those modeled on human annotation instructions, mitigate figurative bias but do not eliminate the tendency to disregard implausibility. LLMs are generally insensitive to prompt phrasing, indicating prediction stability or prompt context disregard.
Quantitative Analysis
The statistical association (χ² and Cramér’s V) between abstractness, plausibility labels, and literalness is significant in human data (p < 0.001, moderate effect sizes for abstractness and literalness). LLMs exhibit only weak associations, with Qwen showing the least correspondence to abstractness. Model accuracy for figurative/literal label prediction remains low (~0.25-0.30) across all LLMs with Kendall’s τ near zero.
Context Generation Nuances
- Human Contextualization: Human annotators frequently alter event constituents or restructure context to achieve plausibility, varying syntactic structure and utilizing broad vocabulary.
- LLM Contextualization: LLMs exhibit syntactic rigidity, repetitive lexical choices (overuse of “event”), and limited constituent alteration even when instructed, producing semantically shallow and often anomalous sentences. When assigned “neither”, models rarely produce valid repairs and tend to ignore instruction on constituent changes.
Implications and Theoretical Considerations
Pragmatic and Computational Impact
The results accentuate the limitations of LLMs in deep semantic plausibility assessment, especially regarding events requiring figurative recontextualization. LLMs’ predilection for defaulting to figurative plausibility signals incomplete modeling of nonsensicality and true semantic anomaly. This finding calls into question the reliability of LLMs in capturing nuanced human interpretations, particularly in generative or interpretive NLP applications where distinction between literal, figurative, and nonsensical must be precise.
Theoretical Insights
The involvement of conceptual abstractness maps (in line with Conceptual Metaphor Theory) demonstrates that humans intuitively leverage abstractness to resolve semantic anomalies, while LLMs, trained on plausible corpora, lack sensitivity to true implausibility and repair mechanisms.
Future Directions
Advancing LLMs’ semantic plausibility modeling will require:
- Enhanced architectures or training objectives incorporating explicit nonsensical and implausible event detection.
- More refined prompt engineering, potentially leveraging structure-sensitive prompts or context-rich annotation protocols.
- Expansion to multi-lingual plausibility studies to probe generality of observed phenomena.
- Integration of probabilistic or multi-label aggregation techniques for improved human-model alignment.
Conclusion
This study rigorously delineates the divergence between human and LLM contextualization of SVO event plausibility and (non-)literalness. Humans excel at intricate sense-making and repair, especially for abstract events, whereas LLMs demonstrate a systematic bias toward plausible/figurative readings and shallow context generation. These findings highlight both practical and theoretical gaps in current LLM-based semantic plausibility modeling, suggesting avenues for future research at the intersection of computational linguistics, pragmatics, and AI system interpretability.