MultiM-Poem: Multimodal Poetry Generation

• MultiM-Poem is a multimodal framework that fuses text, images, audio, and structured data for innovative poetry generation and analysis.
• It employs diverse architectures including end-to-end neural generators, sequential processing, and reinforcement-based adversarial training to optimize poetic quality.
• Leveraging enriched corpora and advanced fusion techniques, MultiM-Poem supports multi-lingual, culturally rich poetry generation, translation, and visual analysis.

MultiM-Poem refers to the class of computational systems, corpora, and methodologies designed to generate, interpret, or align poetry through the integration of multiple modalities, such as text, vision, audio, and structured experience data. Emerging from the fusion of advances in neural language modeling, multimodal reasoning, and structured data alignment, MultiM-Poem systems are characterized by joint or sequential processing of heterogeneous inputs or outputs—including free text, images, audio, and conceptual prompts—for the purpose of poetry generation, translation, visualization, and analysis across a range of cultural and linguistic traditions.

1. Core System Architectures and Modeling Paradigms

MultiM-Poem frameworks are unified by their direct engagement with multi-modal data for poetry generation and analysis, diverging from uni-modal or template-based paradigms. Canonical frameworks employ one or more of the following modeling strategies:

• Fully Neural, End-to-End Generators: As in "Deep Poetry," systems condition neural decoders (e.g., Transformer or hierarchical seq2seq models) on a fused latent space comprising both text and image representations. For instance, Transformer-style generators receive both prefix-encoded user input and CNN-derived visual features, fusing them into a global context vector to seed and modulate poetry generation (Liu et al., 2019).
• Sequential Multi-Channel Processing: Some models, such as experience-inspired architectures, encode time-ordered experiences as paired image/text streams. Visual and textual embeddings are processed via separate RNNs or GRUs, fusing attention-masked outputs per output token. Attention mechanisms are regularized to favor temporally proximate experience-to-line alignment, with fusion achieved through explicit modality weighting (Cao et al., 2022).
• Disentangled/Pipeline Paradigms: Multi-stage approaches process modalities in series, such as: image → thematic label → phrase extraction → poem generation, rather than jointly learning from raw multi-modal pairs. "A Multi-Modal Chinese Poetry Generation Model" exemplifies this with a CNN image-to-theme module, phrase sampling, followed by multi-level RNN decoding constrained by attention across character, phrase, and sentence contexts (Liu et al., 2018).
• Reinforcement and Adversarial Multi-Objective Training: Adversarial or reward-driven objectives integrate modality alignment and poetic style. For example, models employ a combination of multi-modal discriminators for image-poem congruence and style discriminators for free-verse characteristics. Policy gradient methods exploit these critic signals to optimize stochastic generators (Liu et al., 2018).
• Prompt Optimization and Human-in-the-Loop Refinement: Systems such as PoemTale Diffusion incorporate iterative LLM-based multi-stage prompt refinement (MSPR) where LLMs progressively transform poetic segments into vivid and semantically-aligned image prompts, with stopping criteria based on CLIP-derived text-image alignment metrics (Jamil et al., 18 Jul 2025).

2. Data, Preprocessing, and Resource Construction

MultiM-Poem research relies on both newly assembled and enriched corpora covering multiple languages, modalities, and alignments:

• Textual Corpora: Large collections of classical poetry, contemporary lyrics, and annotated verses (e.g., 200,000+ classical Chinese poems, expanded further by ancillary prose) serve as pretraining and fine-tuning material (Liu et al., 2019, Liu et al., 2018).
• Multi-Modal Alignment Datasets: Image-poem pairs (e.g., MultiM-Poem with 8,292 pairs, P4I with 1,111 poems and emotion-entity segmentation) and aligned text/audio corpora (e.g., Shakespeare/Milton corpus with annotations at phoneme, syllable, word, and line) facilitate supervised and contrastively-regularized learning (Liu et al., 2018, Agirrezabal, 26 Jul 2024, Jamil et al., 18 Jul 2025).
• Semantic Enrichment and Annotation: Automated tools and expert-driven labeling are used to construct fine-grained resources—such as phrase taxonomies (ShiXueHanYing), semantic graphs (WordNet-based), and detailed scansion and stress patterns—enabling both controlled generation and downstream analysis (Liu et al., 2019, Jamil et al., 17 Nov 2025, Agirrezabal, 26 Jul 2024).
• Audio and Meter: The aggregation of recitation audio, aligned down to phoneme and syllable, with meter and scansion tags, allows for research at the intersection of prosody, computational metrics, and textual rhythm (Agirrezabal, 26 Jul 2024).

3. Multi-Modal Input Fusion and Generation Mechanisms

MultiM-Poem systems address the technical challenge of fusing heterogeneous modalities—each with distinct statistical and semantic properties:

• Neural Embeddings and Latent Spaces: Encoders map both images (via CNNs or pre-trained vision-language transformers) and text (via RNNs, Transformer encoders, or skip-thought models) into a shared continuous representation. Fusion is achieved by learned linear projections, concatenation, or attention-based pooling, enabling downstream decoders to exploit both visual and conceptual cues in generating each poetic token or line (Liu et al., 2019, Liu et al., 2018, Cao et al., 2022).
• Graph-Based Semantic Structuring: For translation and visualization, semantic graphs constructed over syntactic and lexical units (e.g., lemmatized tokens, WordNet synsets) are clustered via modularity optimization. These clusters determine the structure and content of image generation prompts for diffusion models, ensuring that metaphorical and literal elements are appropriately balanced (Jamil et al., 17 Nov 2025).
• Prompt Refinement Loops and Consistent Attention: In the context of poem-to-image synthesis, iterative prompt refinement leverages LLMs and alignment metrics (Long-CLIP) to distill layered poetic meaning into stages of increasingly image-relevant instructions. Concomitantly, consistent self-attention mechanisms across diffusion batches maintain identity and style coherence across segment-wise images (Jamil et al., 18 Jul 2025).
• User-Driven and Human-in-the-Loop Modes: MultiM-Poem frameworks commonly feature co-writing modes where users specify partial inputs (prefixes, acrostic initials, themes), and the model generates candidate lines subject to structural or poetic constraints using beam search with rule-based filtering (Liu et al., 2019).

4. Training Objectives, Optimization, and Evaluation

Training and evaluation in MultiM-Poem systems are designed to jointly maximize modality alignment, poetic quality, and human interpretability:

• Loss Objectives:
  • Primary Loss: Typically, cross-entropy for next-token prediction.
  • Regularization: L2 weight decay and dropout are standard; some architectures further introduce KL-based priors on attention or curriculum negative sampling to improve modality alignment (Liu et al., 2019, Cao et al., 2022).
  • Adversarial and Reward Signals: Discriminators assign real/generated scores based on cross-modal pairing and poeticness, which are integrated as training rewards in reinforcement learning setups (Liu et al., 2018).
  • Preference Optimization: Translation tasks incorporate Odds Ratio Preference Optimization (ORPO), penalizing the model for insufficient preference for human translations over low-quality alternatives (Jamil et al., 17 Nov 2025).
• Evaluation Metrics:
  • Automatic Metrics: BLEU-1/2/4, METEOR, COMET for translation; CLIP and Long-CLIP for text-image alignment; Distinct-n, Novelty-n, BERTScore, and newly defined measures for diversity, creativity, and relevance (Jamil et al., 17 Nov 2025, Jamil et al., 18 Jul 2025, Cao et al., 2022).
  • Human Evaluation: Experts and AMT raters score generated poems and images on fluency, coherence, theme relevance, aesthetic appeal, semantic alignment, emotional resonance, and overall impression. Studies involve Turing tests, paired preference, and Likert-scale aggregation (Liu et al., 2018, Jamil et al., 18 Jul 2025).
  • Machine Metrics for Prosody: In audio-aligned corpora, F1-scores for syllabification and scansion, as well as correlation of textual length to audio duration, are reported (Agirrezabal, 26 Jul 2024).
• Resource Partitioning and Training Practices: Datasets are generally partitioned into ~70/30 train/validation splits; LoRA-style adaptation and learning rate scheduling are standard in LLM fine-tuning regimes (Jamil et al., 17 Nov 2025). Pipeline training is often sequential, with no end-to-end gradients between modular stages in most legacy systems (Liu et al., 2018).

5. Applications, Corpora, and Practical Deployment

MultiM-Poem advances enable a spectrum of practical use-cases and have motivated the release of several significant multi-modal poetry resources:

Corpus / System	Modalities	Size/Scope	Distinctive features
Deep Poetry (Liu et al., 2019)	Text, images, concepts	200k poems, 3M prose	Multi-modal inputs, human-AI co-writing, mobile UI
MorphoVerse (Jamil et al., 17 Nov 2025)	Low-resource lang. poems, eng.	1,570 poems, 21 languages	ORPO-aligned translations, semantic graph prompts
MultiM-Poem (Liu et al., 2018)	Image, English poems	8,292 image-poem pairs	Joint visual-poetic embedding, adversarial RL
Shakespeare/Milton (Agirrezabal, 26 Jul 2024)	Text, audio, scansion	~12.5 h, 12.6k lines	Four-level text-speech alignment, TEI-XML encoding
P4I (Jamil et al., 18 Jul 2025)	English poems, images	1,111 poems, 798 authors	Entity/emotion tags, MS prompt-refinement, expert eval

• Interactive Generation and Deployment: Systems such as Deep Poetry offer end-to-end user engagement, including WeChat applet interfaces and RESTful backends designed for sub-second latency via model quantization and prompt/result caching (Liu et al., 2019).
• Visual Analytics and Prompt Engineering: Tools like POEM support interactive, visual analytics for prompt optimization, leveraging Sankey diagrams and modular principle recommendations to improve LLM performance on multimodal reasoning tasks (He et al., 6 Jun 2024).
• Translation and Visualization for Minority Languages: The TAI framework demonstrates translation and semantic-graph-based image visualization for morphologically rich, low-resource Indian poetry, supporting inclusive cultural dissemination (Jamil et al., 17 Nov 2025).

6. Limitations, Open Challenges, and Future Directions

Despite recent progress, MultiM-Poem remains an area of ongoing research with several prominent limitations:

• Modal Bottlenecks and Lack of Joint Learning: Many deployed systems rely on staged pipelines (e.g., image-to-theme-to-poetry), with limited end-to-end multimodal parameter sharing. Direct, gradient-based integration of vision and language mappings is uncommon but highlighted as a key direction (Liu et al., 2019, Liu et al., 2018, Jamil et al., 17 Nov 2025).
• Dataset Scale and Diversity Constraints: Most multilingual and multimodal corpora are relatively small (MorphoVerse: 1,570 poems; P4I: 1,111), limiting transfer, robustness, and cultural breadth (Jamil et al., 17 Nov 2025, Jamil et al., 18 Jul 2025).
• Coverage of Genres and Poetic Forms: Certain systems restrict generation to highly structured forms (e.g., 5-/7-character quatrains), with limited support for free verse, narrative poetry, or non-Sinitic/Indic traditions. Broader genre inclusion and explicit style embedding are identified as urgent needs (Liu et al., 2019).
• Qualitative and Aesthetic Metrics: Objective evaluation of poetic quality, affect, and style remains challenging, particularly for text-to-image mappings of layered poetic content. The lack of FID or CLIP-FID reporting, and the need for end-to-end RL frameworks with human-in-the-loop feedback, are open issues (Jamil et al., 17 Nov 2025, Jamil et al., 18 Jul 2025).
• Generalization Beyond Text-Image: Extensions to audio, prosody, gesture, and sensory modalities (as envisioned by the Shakespeare/Milton corpus) have only begun to be explored (Agirrezabal, 26 Jul 2024).

Future prospects include: fully integrated multimodal transformers, dynamic retrieval of experience data, RL-based prompt and structure optimization, multilingual/multicultural alignment, and increased interactivity with human authors for adaptive and culturally nuanced MultiM-Poem generation (Liu et al., 2019, Cao et al., 2022, Jamil et al., 18 Jul 2025, Jamil et al., 17 Nov 2025, Agirrezabal, 26 Jul 2024).