Qwen Storyteller: Script-Aligned Visual Narratives

• Qwen Storyteller is a visual storytelling model lineage built on Qwen2.5-VL 7B that addresses cross-frame entity consistency and referential hallucinations.
• It employs contrastive reinforcement learning alongside structured chain-of-thought reasoning to enhance entity re-identification and narrative coherence.
• The advanced Qwen Storyteller3 stage integrates semantic alignment with authentic script content, improving dialogue attribution and emotional cue accuracy.

Qwen Storyteller is a visual storytelling model lineage built on Qwen2.5-VL 7B for multi-image story generation with explicit grounding, cross-frame entity consistency, and, in a later stage, semantic alignment to movie scripts and subtitles. In the materials associated with this lineage, the earlier system addresses referential hallucinations and entity re-identification through contrastive reinforcement learning, while Qwen Storyteller3 extends the same trajectory by conditioning generation on aligned screenplay dialogue, character names, and emotional cues derived from the StoryMovie dataset. The resulting progression is presented as a shift from visual grounding alone toward semantically grounded narrative generation that aims to reduce incorrect dialogue attribution, character interaction errors, and fabricated affective content (Oliveira et al., 9 Jul 2025, Oliveira et al., 25 Feb 2026).

1. Model lineage and problem setting

Qwen Storyteller is situated in visual storytelling, where a model must generate a coherent narrative from a sequence of images rather than captioning each frame independently. The central failure mode identified for such systems is not merely object or character detection error, but the inability to maintain correct identity, referential continuity, and higher-level semantic relations across frames. The earlier work states that visual storytelling systems, particularly large vision-LLMs, often fail to recognize when entities in different images represent the same individuals or objects, which leads to inconsistent references and referential hallucinations. The later work further narrows the problem by observing that even models that correctly ground entities in images may still hallucinate semantic relationships, including incorrect dialogue attribution, character interactions, or emotional states (Oliveira et al., 9 Jul 2025, Oliveira et al., 25 Feb 2026).

The lineage described in the source materials has three distinct capability layers. First, there is structured Chain-of-Thought reasoning and cross-frame entity grounding inherited from StoryReasoning. Second, there is contrastive RL-based entity re-identification, described in the technical summary as inherited from Storyteller2. Third, there is semantic alignment with authentic script content, specifically character names, dialogue, and emotional cues. This progression is important because it distinguishes low-level visual grounding from narrative fidelity. A plausible implication is that the authors treat grounded storytelling as a composite problem with at least three separable constraints: frame-level perception, cross-frame identity persistence, and script-level semantic coherence.

A concise summary of the progression described in the materials is given below.

Stage	Base model or input	Primary capability
Qwen Storyteller	Qwen2.5-VL 7B	Entity re-identification in visual storytelling via contrastive reinforcement learning
Qwen Storyteller3	Qwen2.5-VL 7B + StoryMovie	Semantic alignment with authentic script content
StoryMovie pipeline	Movie scripts + subtitles + MovieNet frames	Dialogue attribution by linking character names to temporal positions

2. Contrastive reinforcement learning and entity re-identification

The earlier Qwen Storyteller training stage is defined around a contrastive reinforcement learning framework that teaches the model when to establish entity connections across frames. The starting point is the Story Reasoning dataset with $N = 4\,178$ real stories, each containing between 5 and 15 frames. An equal number of synthetic negative stories, also $4\,178$, are generated by deterministic sampling across unrelated movies, yielding a real-to-synthetic ratio of $1:1$. For a synthetic story index $s$ and desired frame count $n$, the $i$-th frame is chosen by

$$\text{story\_idx}_i = (s \times 17 + i \times 31) \bmod N$$

and

$$\text{img\_idx}_i = (s + i \times 7) \bmod |I_{\text{story\_idx}_i}|.$$

This construction directly operationalizes incoherent sequences as negatives for training (Oliveira et al., 9 Jul 2025).

The optimization framework uses Direct Preference Optimization over offline preference pairs extracted via a dual-component reward $R$. The reward is

$$R(c,s,\mathbb{I},r)= \begin{cases} 0.5 \cdot R_{\text{reid}}(c,r)+0.5 \cdot R_{\text{ground}}(s) & \text{if structurally valid} \ -1.0 & \text{otherwise} \end{cases}$$

where $4\,178$0 is the Chain-of-Thought table, $4\,178$1 the generated story, $4\,178$2 the input images, and $4\,178$3 indicates whether the sequence is real or synthetic. The entity re-identification reward is

$4\,178$4

with $4\,178$5 and $4\,178$6. The character and object persistence scores are

$4\,178$7

The grounding reward is

$4\,178$8

with $4\,178$9 and $1:1$0. Structural invalidity is treated harshly: any output with missing <gdi> tags, wrong entity IDs, malformed tables, out-of-bounds boxes, or missing narrative phases receives $1:1$1 (Oliveira et al., 9 Jul 2025).

Preference pairs are formed from sampled candidates $1:1$2 for each input $1:1$3, using a margin criterion $1:1$4. The DPO loss is

$1:1$5

with $1:1$6. Training uses Qwen2.5-VL 7B with LoRA rank 2048 and $1:1$7 on the self-attention layers, AdamW with learning rate $1:1$8, weight decay 0, batch size 8 image sequences per update, and 3 training epochs. No on-policy RL steps are used; the procedure is entirely offline DPO (Oliveira et al., 9 Jul 2025).

Empirically, this stage improves grounding mAP from 0.27 to 0.31, F1 from 0.35 to 0.41, METEOR from 0.14 to 0.17, ROUGE-L from 0.16 to 0.18, BLEU-4 from 0.054 to 0.057, and combined cross-frame persistence for entities in 5 or more frames from 29.3% to 33.3%. Pronoun grounding accuracy improves across all reported pronoun types except “its,” and well-structured stories rise from 79.1% to 97.5%. These results establish the re-identification stage as a foundation for later semantic alignment rather than a complete solution to narrative hallucination (Oliveira et al., 9 Jul 2025).

3. StoryMovie and script-subtitle alignment

The StoryMovie dataset is introduced to address a different layer of error: semantically incorrect stories generated from visually grounded inputs. StoryMovie consists of 1,757 multi-image stories inherited from StoryReasoning and drawn from a subset of MovieNet movies with both scripts and subtitles available. The train/test split is 1,494 training and 263 test, corresponding to 85%/15%. Reported story statistics are a word count of $1:1$9, entity references per story of 119.54, character mentions of 32.64, object references of 4.70, setting descriptions of 32.65, action references of 49.55, and 9.21 distinct characters (Oliveira et al., 25 Feb 2026).

The alignment pipeline is based on token-level Longest Common Subsequence matching between screenplay dialogue and subtitle text. Given token sequences $s$0 and $s$1, the LCS table $s$2 is defined by

$s$3

Once a maximal subsequence is found, matches are extended bidirectionally until either the speaker in the script changes or a scene break occurs, such as a new slug line. Each aligned block inherits the precise subtitle timestamp. This is the core mechanism that transfers character names and emotional cues from scripts into a temporally anchored representation (Oliveira et al., 25 Feb 2026).

The final step links aligned script-subtitle segments to image frames using MovieNet annotations that provide frame-level time ranges. For each story, the pipeline selects script-subtitle segments whose timestamps overlap the story’s frame interval. Character names from the script and dialogue tokens are thereby time-synchronized to each image frame, enabling downstream models to ground “char1” tags in actual character names, such as “Mary,” and exact utterances. In qualitative terms, the dataset is designed to convert abstract grounding tags into semantically specified references.

This dataset design directly targets a common misconception in multimodal narrative generation: that correct entity localization is sufficient for faithful storytelling. The StoryMovie construction indicates that grounding tags without script-level alignment may still permit fabricated utterances, generic relationship descriptions, and unsupported emotional inferences. The explicit synchronization of names, utterances, and timestamps is therefore not auxiliary metadata but part of the supervision signal.

4. Qwen Storyteller3 architecture and supervised fine-tuning

Qwen Storyteller3 is described as a fine-tuned model built on Qwen2.5-VL 7B. Its inherited capabilities are structured Chain-of-Thought reasoning, cross-frame entity grounding from StoryReasoning, and contrastive RL-based entity re-identification from Storyteller2. Its new capability is semantic alignment with authentic script content, specifically character names, dialogue, and emotional cues (Oliveira et al., 25 Feb 2026).

Training is performed by supervised fine-tuning on StoryMovie with the standard token-level cross-entropy loss

$s$4

where $s$5 is the ground-truth story token at position $s$6, $s$7 is the image sequence, $s$8 is the structured CoT, and $s$9 is the aligned script+subtitle segment. The technical summary explicitly states that no additional alignment regularizer was introduced, because semantic grounding arises naturally from conditioning on $n$0. This detail distinguishes the method from approaches that impose explicit auxiliary constraints or consistency penalties (Oliveira et al., 25 Feb 2026).

The reported hyperparameters are LoRA rank $n$1 with $n$2, AdamW with decoupled weight decay, cosine annealing with peak learning rate $n$3, effective batch size 32 via gradient accumulation, and 3 epochs. In methodological terms, the contrast with the earlier DPO stage is notable. The re-identification phase is preference-based and contrastive; the semantic-alignment phase is supervised and conditional on externally aligned textual evidence. A plausible implication is that the authors view semantic correction as achievable through improved conditioning rather than through a separate reward model.

The conditioning tuple $n$4 formalizes the model’s intended information flow: image evidence $n$5, structured narrative reasoning $n$6, and aligned screenplay-subtitle evidence $n$7. This arrangement suggests that semantic errors such as incorrect speaker attribution are not treated as post hoc correction problems but as failures of the conditioning context available during generation.

5. Evaluation protocols and empirical results

The two stages are evaluated with different protocols. The earlier entity re-identification work uses grounding precision, recall, F1, 11-point interpolated average precision per story averaged across all stories, pronoun grounding accuracy, cross-frame entity persistence, and standard language metrics including METEOR, ROUGE-L, and BLEU-4. The later semantic-alignment work evaluates on 341 held-out stories using pairwise preference tests judged by DeepSeek V3 with three reference types: subtitles for dialogue attribution, description for visual scene understanding, and synopsis for broader narrative alignment. In the later protocol, each story pair is evaluated in three independent annotation runs, under blind comparison with randomized order, with reference snippets provided according to the evaluation type, and consensus win rates aggregated as mean ± std (Oliveira et al., 9 Jul 2025, Oliveira et al., 25 Feb 2026).

Against base Qwen2.5-VL 7B, Qwen Storyteller3 achieves 89.9% ± 1.4 versus 3.5% ± 0.8 on subtitles, 63.4% ± 0.9 versus 4.1% ± 0.9 on description, and 87.6% ± 1.1 versus 6.8% ± 0.7 on synopsis. Against Qwen Storyteller trained without script alignment, it achieves 48.5% ± 1.4 versus 38.0% ± 1.6 on subtitles, 35.5% ± 1.7 versus 15.2% ± 1.3 on description, and 42.7% ± 1.5 versus 28.5% ± 1.4 on synopsis. The technical summary states that these results confirm that semantic alignment dramatically boosts dialogue attribution accuracy and improves overall narrative grounding (Oliveira et al., 25 Feb 2026).

The comparison between the earlier and later systems is especially informative. The contrastive RL stage improves identity persistence, pronoun grounding, and structural validity, while the StoryMovie stage improves dialogue attribution and broader narrative alignment. This suggests a layered decomposition of evaluation: mAP, F1, and persistence capture whether entities are tracked and referenced consistently, whereas subtitle and synopsis preferences capture whether those grounded entities participate in the correct narrative relations. The reported 48.5% versus 38.0% comparison against the script-agnostic storyteller is the clearest evidence that visual grounding alone does not resolve higher-level semantic hallucinations (Oliveira et al., 9 Jul 2025, Oliveira et al., 25 Feb 2026).

6. Qualitative gains, limitations, and interpretive significance

The later work reports three qualitative gains. First, in dialogue attribution, Qwen Storyteller3 correctly assigns lines from the aligned script, exemplified by “Mary: ‘I won’t let you go’,” to the visual entity tag <gdc=char3/>, replacing fabricated utterances. Second, in relationship mapping, scenes depicting a family dinner are described with correct kinship, such as “a tense mother–daughter exchange,” rather than generic emotional narration. Third, emotional cues such as “(whispering)” from the script appear in the generated story, yielding richer affective descriptions. These examples are presented as evidence that alignment to authentic screenplay content changes not only lexical choice but also the model’s representation of interpersonal and affective structure (Oliveira et al., 25 Feb 2026).

The reported limitations are equally specific. In scenes with sparse subtitles or mis-timed subtitles, the model reverts to visual-only inference and may fabricate generic dialogue. The pipeline and model are attuned to professional cinema, and performance on informal or user-generated imagery, including home videos, remains untested. The evaluation also inherits potential bias from DeepSeek V3 as an LLM judge, and linguistic biases in such judges may affect preference judgments. These limitations constrain the interpretation of the reported gains: improvements are established within a temporally aligned, movie-domain setting, and not yet across arbitrary multimodal narrative corpora (Oliveira et al., 25 Feb 2026).

Taken together, the materials portray Qwen Storyteller as a staged research program rather than a single static model. The earlier stage teaches the model to establish or withhold cross-frame identity links appropriately; the later stage supplies external semantic evidence that ties those grounded entities to authentic names, utterances, and relationship cues. The conclusion of the StoryMovie work states that this progressive training, from visual grounding in StoryReasoning to semantic alignment, establishes a blueprint for future multimodal narrative models capable of combining visual, temporal, and script-based signals for faithful story generation. A plausible implication is that the central research contribution lies in separating and then recombining perceptual grounding, entity persistence, and script-conditioned semantics as distinct training problems (Oliveira et al., 25 Feb 2026).