POEMetric: A Framework for Poetry Evaluation
- POEMetric is a comprehensive framework that integrates rule-based checks, LLM judging, and expert validation to assess both formal correctness and creative literary qualities.
- It distinguishes three performance layers—basic instruction-following, advanced creative abilities, and overall poem quality—highlighting LLM strengths in form but deficits in nuanced artistry.
- The evaluation employs a curated corpus of 203 fixed-form poems, utilizing metrics like MATTR and repetition rate to quantify lexical diversity and originality.
Searching arXiv for the POEMetric paper and closely related poetry-evaluation work. arXiv query: (Li et al., 4 Apr 2026) POEMetric; (Wang et al., 29 Jun 2026) Poller; (Kranti et al., 9 Oct 2025) MetricalARGS. POEMetric is a comprehensive framework for poetry evaluation that was introduced to compare human-written poems and poems generated by LLMs across formal correctness, prompt adherence, advanced poetic abilities, and global literary appraisal (Li et al., 4 Apr 2026). It was designed around a specific deficiency in earlier evaluation practice: low-level checks such as rhyme and meter, or broad text-quality measures such as fluency and coherence, do not adequately capture qualities that literary criticism treats as central to poetry, including creativity, idiosyncrasy, emotional resonance, imagery, and the use of literary devices. In its original formulation, POEMetric combines a curated human poetry corpus, rule-based evaluation, LLM-as-a-judge, and human expert validation to ask how close LLMs are to human poets under matched formal and thematic conditions (Li et al., 4 Apr 2026).
1. Definition, scope, and rationale
POEMetric was defined as a unified evaluation ecosystem rather than a single scalar score. Its purpose is to compare human poets and LLMs on the same tasks while separating three layers of performance: basic instruction-following, advanced creative abilities, and general appraisal (Li et al., 4 Apr 2026). This design reflects the observation that a poem may satisfy a requested fixed form and theme yet still fall short as poetry in the stronger literary sense.
The framework was motivated by fragmentation in computational poetry evaluation. Prior work often isolated formal properties such as meter and rhyme, or relied on ad hoc human judgments, or used LLM-as-a-judge without systematic validation. POEMetric instead integrates objective rule-based checks for form, quantitative lexical measures, and model-based judgments that were explicitly compared against human experts (Li et al., 4 Apr 2026).
The term should also be distinguished from unrelated arXiv usages of similar acronyms. It is separate from “POEM” in recommendation systems (Che et al., 29 Jun 2026), “POEM” in reinforcement learning (Czworkowski et al., 21 Jan 2026), and the metrological interpretation of “POEMetric” associated with SR-POEM in a sounding-rocket Weak Equivalence Principle experiment (Phillips et al., 2010). In contemporary poetry-evaluation research, however, “POEMetric” most directly denotes the 2026 framework for assessing poetic generation quality (Li et al., 4 Apr 2026).
2. Evaluation dimensions and conceptual structure
POEMetric organizes evaluation into ten dimensions grouped under three categories: basic instruction-following abilities, advanced creative abilities, and general appraisal (Li et al., 4 Apr 2026).
| Category | Dimensions |
|---|---|
| Basic instruction-following | Form Accuracy; Theme Alignment |
| Advanced creative abilities | Creativity; Lexical Diversity; Idiosyncrasy; Emotional Resonance; Imagery; Literary Devices |
| General appraisal | Overall Poem Quality; Authorship Estimation |
The first category addresses compliance with the prompt. Form Accuracy concerns whether the poem follows the specified fixed form, including meter and rhyme pattern. Theme Alignment measures how well the poem reflects the requested theme. In POEMetric’s experiments, these dimensions test whether a model can satisfy explicit formal and semantic constraints rather than merely generate plausible verse (Li et al., 4 Apr 2026).
The second category addresses what the framework treats as core literary qualities. Creativity concerns novelty, originality, and imagination; Lexical Diversity concerns vocabulary variation; Idiosyncrasy concerns stylistic distinctiveness and recognizable voice; Emotional Resonance concerns whether the poem evokes feeling in the reader; Imagery concerns vivid sensory evocation; and Literary Devices concerns the use of simile, metaphor, personification, and allusion. These dimensions correspond to the framework’s central claim that successful poetry evaluation must move beyond prosodic correctness and generic fluency (Li et al., 4 Apr 2026).
The final category captures holistic judgment. Overall Poem Quality asks whether the text succeeds as a poem in aggregate, while Authorship Estimation asks whether it appears human-written. A plausible implication is that POEMetric treats indistinguishability from human poetry not as the sole criterion of success, but as one global symptom of broader poetic competence.
3. Corpus construction and experimental protocol
The human reference corpus in POEMetric contains 203 English poems spanning roughly 200 years, collected from Poetry Foundation and the Academy of American Poets after an initial pool of 1,309 poems was filtered to retain texts that clearly follow a consistent meter and rhyme pattern (Li et al., 4 Apr 2026). The dataset covers seven fixed forms: 95 ballads, 9 ghazals, 6 limericks, 3 pantoums, 7 sestinas, 71 sonnets, and 12 villanelles. Each poem is annotated with form, meter pattern, rhyme pattern, theme, and imagery (Li et al., 4 Apr 2026).
These annotations were used directly in the generation prompt. The standard prompt asked a model to write an original English poem subject to four fields: form, meter, rhyme, and theme, and to return only the poem. Thirty LLMs from seven organizations were evaluated under this protocol, yielding 6,090 LLM poems from the same 203 prompt specifications used for the human corpus (Li et al., 4 Apr 2026). This controlled pairing makes the comparison unusually strict: humans and models are effectively solving the same poetic assignment.
The corpus design also constrains the framework’s scope. Because the benchmark is limited to fixed forms, POEMetric emphasizes formal adherence that can be operationalized through rule-based checks. This suggests that the framework is strongest when the poetic tradition provides explicit structural targets, and less directly applicable to free verse without further adaptation.
4. Rule-based scoring, LLM judging, and validation
POEMetric evaluates form through a rule-based pipeline that tokenizes each poem into lines and words, extracts stress patterns and rhyme information using a pronouncing dictionary such as CMUdict, and then compares the observed structure to the target specification (Li et al., 4 Apr 2026). For several forms with distinctive nonlocal constraints, the evaluation includes form-specific checks: ghazal couplets and radif/qaafiya, sestina end-word permutations and envoi, villanelle refrains, pantoum line repetitions, and limerick rhyme patterns. Meter and rhyme are accepted with a 70% match threshold to tolerate limited variation rather than enforce absolute identity (Li et al., 4 Apr 2026).
Two additional rule-based measures complement structural evaluation. Lexical Diversity is quantified with Moving Average Type-Token Ratio (MATTR), computed by sliding a fixed-size window over the text and averaging type-token ratios across windows. Creativity, in one quantitative sense, is approximated by repetition rate relative to the human source poem: greater lexical overlap indicates greater imitation and therefore lower originality (Li et al., 4 Apr 2026).
For subjective dimensions, POEMetric uses Gemini-2.5-Pro as the sole LLM judge. The selection was empirical. In a pilot comparison against human experts, Gemini-2.5-Pro achieved observed agreement , while GPT-4o reached and DeepSeek-R1 reached (Li et al., 4 Apr 2026). Gemini also showed score variance closer to that of human judges, whereas GPT-4o and DeepSeek-R1 produced flatter and more optimistic ratings. Human validation involved seven literary experts with backgrounds in poetry or English literature, and the same questionnaire was used for both human and LLM judging (Li et al., 4 Apr 2026).
5. Empirical findings: formal competence and poetic deficit
POEMetric’s main empirical result is asymmetric. LLMs are strong at basic instruction-following, but they remain behind humans on advanced poetic abilities and overall poem quality (Li et al., 4 Apr 2026). The top model achieved a Form Accuracy score of 4.26 out of 5.00 and a Theme Alignment score of 4.99 under Gemini-2.5-Pro judging, indicating that formal control and prompt compliance are no longer the primary bottlenecks for the best systems (Li et al., 4 Apr 2026).
The advanced abilities show a different profile. Human poets achieved Creativity 4.02, Idiosyncrasy 3.95, Emotional Resonance 4.06, Imagery 4.49, and Literary Devices 4.67, while no model matched those levels across the board (Li et al., 4 Apr 2026). Humans also outperformed the best LLM in Overall Poem Quality, scoring 4.22 against the best model’s 3.20 (Li et al., 4 Apr 2026). These results support the framework’s central thesis that contemporary LLMs can emulate poetic form and theme without matching the distinctiveness, emotional force, or literary subtlety of human poets.
The results are not uniformly unfavorable to LLMs. In rule-based lexical statistics, some LLMs exceeded humans on MATTR, indicating higher measured vocabulary variation (Li et al., 4 Apr 2026). Yet those same systems often exhibited elevated repetition relative to the corresponding human originals, which the framework interprets as evidence of imitation or pastiche rather than deeper originality. This combination is important: lexical diversity alone is not treated as sufficient evidence of creativity.
The authorship-estimation results further reinforce the human–model gap. Both LLM and human judges could generally distinguish human poems from LLM outputs, and Gemini-2.5-Pro explicitly recognized 80 of the 203 human poems as existing poems or stylistically identifiable works (Li et al., 4 Apr 2026). This suggests that current LLM poetry is not yet indistinguishable from human poetry, even when it satisfies fixed-form constraints well.
6. Relation to adjacent frameworks and methodological significance
POEMetric occupies a particular place in the broader literature on computational poetry. It is narrower than task taxonomies such as METRICALARGS, which organizes poetry-related NLP into Analysis, Retrieval, Generation, and Support rather than defining a single evaluation benchmark (Kranti et al., 9 Oct 2025). It is also distinct from Poller, which evaluates poetry understanding rather than poetry generation and does so through poet-aware, role-conditioned LLM judging across eight interpretive dimensions (Wang et al., 29 Jun 2026). Taken together, these works suggest a broader movement toward multidimensional, poetry-specific evaluation rather than reliance on generic NLP metrics.
At the same time, POEMetric inherits and extends older traditions of explicit formal evaluation. Earlier work on machine sonnet generation enforced hard constraints for rhyme and iambic pentameter and effectively treated structural validity as a primary metric (Benhardt et al., 2018). PROPOE 2 similarly operationalized poem assembly through explicit rhythmic and phonological scores such as Jaccard similarity over stress positions, rhyme-type matching, and internal-rhyme measures (Sousa et al., 2024). POEMetric differs by placing those formal concerns inside a larger evaluative frame that includes creativity, imagery, idiosyncrasy, and human-likeness (Li et al., 4 Apr 2026).
The framework’s stated limitations are also substantive. It is restricted to English, and only to seven fixed forms; free verse is outside its current scope. The human corpus is modest in size, many evaluation dimensions remain inherently subjective, and the chosen LLM judge may share biases with academic literary criticism or recognize canonical poems from pretraining (Li et al., 4 Apr 2026). This suggests that POEMetric is best understood not as a final universal metric, but as a benchmarked evaluation architecture: one that separates formal competence from poetic artistry and thereby makes the remaining human–LLM gap more analytically visible.