Improving Visual Storytelling with Multimodal Large Language Models

Abstract: Visual storytelling is an emerging field that combines images and narratives to create engaging and contextually rich stories. Despite its potential, generating coherent and emotionally resonant visual stories remains challenging due to the complexity of aligning visual and textual information. This paper presents a novel approach leveraging LLMs and large vision-LLMs (LVLMs) combined with instruction tuning to address these challenges. We introduce a new dataset comprising diverse visual stories, annotated with detailed captions and multimodal elements. Our method employs a combination of supervised and reinforcement learning to fine-tune the model, enhancing its narrative generation capabilities. Quantitative evaluations using GPT-4 and qualitative human assessments demonstrate that our approach significantly outperforms existing models, achieving higher scores in narrative coherence, relevance, emotional depth, and overall quality. The results underscore the effectiveness of instruction tuning and the potential of LLMs/LVLMs in advancing visual storytelling.

• The paper introduces a novel framework combining LLMs, LVLMs, and reinforcement learning to address visual storytelling challenges.
• It employs explicit instruction tuning and curated datasets to enhance narrative coherence, character continuity, and emotional engagement.
• Empirical results show significant improvements over baselines, with higher scores in coherence, relevance, and overall narrative quality.

Improving Visual Storytelling with Multimodal LLMs: A Technical Summary

Problem Formulation and Motivation

Visual storytelling, the generation of coherent narratives from image sequences, presents challenges in maintaining temporal coherence, character continuity, and emotional resonance. The alignment between visual and textual modalities is nontrivial, requiring advanced multimodal integration and narrative reasoning capabilities. Conventional metrics for evaluation fail to capture narrative depth, further complicating progress measurement in this domain.

This paper introduces a comprehensive research framework leveraging both LLMs and large vision-LLMs (LVLMs), augmented by instruction tuning and reinforcement learning, to address the aforementioned difficulties. The approach is grounded in novel data curation and multimodal learning strategies tailored for visual story generation.

Methodology

Dataset Construction

A new dataset is curated, comprising diverse visual story sequences from comics, illustrated books, and educational materials. Each sequence is annotated with event-driven captions encapsulating actions, emotional states, and context. Additionally, multimodal resources including video clips with aligned textual narratives are incorporated to capture temporal and scene dynamics, facilitating richer model understanding.

Instruction Tuning Paradigm

The paper develops a set of explicit instruction tasks for robust multimodal story generation:

• Caption Generation: Individual image descriptions with semantic granularity.
• Story Continuation: Sequential narrative development conditioned on earlier context.
• Character/Scene Consistency: Maintaining visual and textual coherence across the story arc.
• Emotion/Context Recognition: Embedding affective and situational information.

Instruction tuning acts as a modular supervisory signal to enhance model generalization and narrative integration.

Training Protocol: Supervised and Reinforcement Learning

The training regimen is a hybrid of supervised and reinforcement learning:

• Supervised Pretraining: The negative log-likelihood objective is employed over image-text pairs, establishing a baseline architecture capable of generating contextually relevant narratives.
• Reinforcement Learning with GPT-4 Reward Signal: A reward function, instantiated via GPT-4, quantifies narrative quality (coherence, relevance, emotional depth). The expected reward is maximized using model-generated samples, allowing fine-grained feedback for narrative enhancement.

The combined loss is $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda \mathcal{L}_{\text{RL}}$$, with $\lambda$ as a balancing hyperparameter.

Empirical Evaluation

Quantitative Metrics

The model is benchmarked against established LVLMs (Qwen-VL, MiniGPT-4, LLaVA-1.5 7B), using GPT-4-based evaluation criteria. The method demonstrates strong numerical advances:

• Coherence: 8.9 (vs. 8.2 for LLaVA-1.5 7B)
• Relevance: 8.7 (vs. 7.9)
• Emotional Depth: 8.5 (vs. 7.7)
• Overall Quality: 8.7 (vs. 8.0)

Qualitative Human Assessment

Human judges rate the model consistently higher than baselines, reaffirming improvements in narrative fluency, character development, and affective engagement. Human evaluations complement GPT-4 metrics, establishing the model’s practical viability in real-world storytelling scenarios.

Ablation Analysis

Ablation studies underscore the critical influence of instruction tuning and reinforcement learning. Removal of instruction tuning degrades performance significantly (e.g., coherence drops to 7.5), validating the necessity of explicitly formulated supervisory objectives for advanced multimodal integration.

Narrative Quality and Analytical Breakdown

Detailed examination reveals notable advances in character development, plot progression, and emotional engagement. The model maintains strong plot intentionality and inter-frame consistency, outperforming baselines across all measured aspects.

Implications and Future Directions

The integration of LLMs, LVLMs, and instruction tuning forms a robust foundation for visual storytelling, enabling the automatic creation of narratives with contextual granularity and affective depth. This approach is extensible to other domains requiring multimodal reasoning, including education, interactive media, and digital content creation.

Theoretical implications include advancing the paradigm of multimodal instruction tuning, reinforcing the necessity of explicit task formulations for emergent narrative capabilities. Practically, the use of GPT-4 as a reward signal offers a scalable evaluation framework, potentially generalizable to domains beyond storytelling.

Future research can explore the incorporation of additional modalities (audio, gesture), further refinement of reinforcement learning reward semantics, and domain-specific narrative customization. The methodology provides a foundation for multimodal narrative agents capable of sophisticated story generation in increasingly complex contexts.

Conclusion

This research presents a comprehensive approach to visual story generation, leveraging advanced multimodal architectures and explicit instruction tuning combined with reinforcement learning. Empirical results, both quantitative and qualitative, substantiate the superiority of the proposed method in generating coherent, relevant, and emotionally resonant narratives. The study highlights the pivotal role of instruction-driven learning strategies and offers promising avenues for further multimodal AI research and practical deployment.