Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

Abstract: Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging LLMs to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.

• The paper introduces Poller, a role-based evaluator that integrates authorial context to reduce evaluation error, notably cutting GPT-4-Turbo’s error from 14.38 to 6.13 (57.37% improvement).
• It employs a multidimensional scoring framework across eight literary dimensions, achieving error reductions of up to 94.55% in areas like Rhetorical Techniques and Defamiliarization.
• The study demonstrates that tailored prompting strategies can mitigate LLM evaluation biases, providing a scalable, low-cost proxy for human poetry interpretation.

Poller: Evaluating LLM Suitability for Poetry Understanding Tasks

Introduction

Automated evaluation of poetry remains a critical open problem in natural language processing, particularly due to the limitations of existing metrics and the inherently subjective and multidimensional nature of poetry interpretation. While LLMs have shown promise in evaluative roles for various NLP tasks, their reliability on creative domains—especially for nuanced genres such as modern Chinese poetry—has not been established. "Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?" (2606.30556) systematically investigates LLMs' effectiveness in poetry understanding evaluation and proposes Poller, a role-based evaluator paradigm designed to bridge the gap between automated and human expertise in this context.

Limitations of Automatic and LLM-based Poetry Evaluation

Conventional automatic metrics (e.g., BLEU, ROUGE) are suboptimal for poetry due to their inability to model higher-order literary features like figurative language, rhythm, and the underlying cultural context. Human evaluation is considered the gold standard but creates scalability and reproducibility challenges. The study quantifies this gap by prompting leading LLMs (GPT-4-Turbo, DeepSeek-R1, Qwen2.5-Turbo, Claude3.5-sonnet) to assess poem interpretations across eight expert-defined dimensions: Content, Language, Imagery, Rhetorical Techniques, Rhythm, Defamiliarization, Thought/Emotion, and Modernity. Baseline evaluations (direct LLM judgment) generated error distributions significantly different from human evaluations, with LLMs universally assigning higher scores and failing to distinguish nuances, especially for abstract dimensions—validating the lack of reliability in naive LLM evaluative use.

The Poller Method: Role-based Author Emulation

Poller is introduced as a meta-prompting framework requiring LLMs to assume the role of a poem's author by integrating biographical data, authorial stance, and curated external critique directly into the LLM's context window. The hypothesis is that injecting this layered authorial context anchors the LLM in the poet’s interpretive perspective, thereby reducing mismatches with human evaluators (in this case, the original authors). Particularly, Poller operates by:

• Providing models with structured author information, including biography ($I$), poet's own views/theory ($V$), and third-party evaluations ($C$).
• Tasking LLMs with multidimensional evaluation (Content, Language, etc.) of poem interpretations, scoring each along a continuous scale (0–100) according to the same rubric used by poet-evaluators.
• Employing discrete prompt variants ($P_0$ to $P_4$), with $P_3$ incorporating explicit role declaration and simulation of unpublished poem context (Figure 1).

  Figure 1: Poller’s modular workflow, with the role-playing evaluator at the core, integrating multidimensional scoring and authorial context.

Experimental Results and Quantitative Findings

Experiments leverage 40 high-quality modern Chinese poems authored by five experts, systematically evaluated both by LLMs and the original poets. Strong numerical outcomes of Poller include:

• Error Reduction: Role-based Poller dramatically reduces absolute evaluation error ($|$LLM – human$|$); for GPT-4-Turbo, average error falls from 14.38 ($P_0$) to 6.13 ($P_3$) ($V$057.37%), indicating error rates within a plausible margin for human-like judgment.
• Dimension-specific Impact: For Rhetorical Techniques and Defamiliarization, Poller-based LLMs exhibit 94.55% and 89.53% error reductions, respectively. These results are unobtainable under default LLM prompting.
• Prompt Sensitivity: Prompting strategy significantly modulates efficacy. Explicit role affirmation and context (as in $V$1 and $V$2) enhance LLM evaluation accuracy most consistently across models and dimensions.
• LLM Model Comparison: While all models benefit, the magnitude of gain varies; DeepSeek-R1 and GPT-4-Turbo demonstrate the largest improvements, with Qwen2.5-Turbo and Claude3.5-sonnet showing lower but still substantial advances.

  Figure 2: A concrete example of author-role Poller interaction and multidimensional scoring for a complex poem.

Methodological Analysis and Prompt Ablation

Ablation studies indicate that separating evaluation prompts by dimension yields marginally higher scores in isolated tasks (e.g., scoring Rhetorical Techniques alone), though holistic, multidimensional evaluation remains the operational focus due to the inherently cross-dimensional nature of poetry interpretation. Poller’s design enables flexible adaptation of evaluator identity and context, suggesting broader applicability with available high-quality metadata.

Implications and Theoretical Significance

Poller’s empirical results validate that role-based author emulation fundamentally alters and improves LLM evaluation behavior for creative tasks. This approach moves LLM-based evaluation beyond surface lexical matching or genre pattern recognition by leveraging contextual prompts that encode the cultural and interpretive frame of the literary creator. Practically, this enables scalable, low-cost, and reproducible evaluations of poetry understanding—critical for both annotation and benchmarking large-scale literary datasets, especially as new LLMs are introduced.

Theoretically, these findings imply that contextually grounded, perspective-conditioned LLMs can, under certain conditions, function as reliable proxy-evaluators for tasks with high subjective variability. This generalizes the growing body of research on persona and role-conditioning in LLMs to rigorous evaluative settings and demonstrates that with suitable prompt engineering, LLMs’ evaluation biases and over-positivity can be mitigated in domains previously considered unreproducible or intractable for automation.

Future Directions

Immediate extensions would involve validating Poller across other poetry genres (classical, translated works) and languages, measuring its robustness under distribution shift and cross-cultural context. Furthermore, automating or semi-automating the assembly of the authorial context package would improve Poller’s accessibility for large-scale deployment and for texts where living poet context is unavailable. Integration with reinforcement learning or targeted tuning—grounded in the author-conditioned framework—represents a further avenue for minimizing the residual modeling-evaluator gap. Additionally, spectral analysis of score alignment and human/LLM evaluation clustering could provide deeper insight into the interpretive variance captured by role-conditioned LLMs.

Conclusion

The role-based Poller evaluator method demonstrates that with sufficiently detailed authorial and contextual information, LLMs can reliably mimic human evaluators in poetry understanding tasks. This approach addresses the inadequacy of automatic metrics for literary domains and, by minimizing evaluation error, notably in the most challenging aspects of poem interpretation, offers a scalable solution that hybridizes human expertise with LLM efficiency and consistency. Poller establishes a methodological precedent for leveraging intrinsic authorial perspective in LLM evaluation for creative and highly subjective NLP tasks, providing a practical blueprint for future evaluative frameworks in literary AI.