ImageStoryGen-500K: Visual Story Benchmark

• The paper introduces ImageStoryGen-500K, a benchmark that evaluates image-conditioned creative generation by converting images and nuanced instructions into long-form texts.
• It leverages a pipeline with structured visual extraction and prompt generation to capture detailed cues, ensuring outputs adhere to specific styles and narratives.
• Evaluations using VisuCraft metrics show significant gains in creativity and instruction adherence, underlining the benchmark's challenge and impact.

Searching arXiv for the benchmark and closely related visual story-generation work. ImageStoryGen-500K is a self-constructed benchmark introduced in “VisuCraft: Enhancing Large Vision-LLMs for Complex Visual-Guided Creative Content Generation via Structured Information Extraction” (Jiang et al., 4 Aug 2025). It is designed for evaluating complex visual-guided creative generation rather than for pretraining. At the dataset level, it contains 500,000 diverse images paired with creative generation instructions, and its task format is image $\rightarrow$ instruction $\rightarrow$ long-form creative text. The benchmark targets settings in which a model must convert an image plus a nuanced instruction into a visually grounded and stylistically controlled response, with evaluated task types including story generation, poetry composition, and advertising copy generation (Jiang et al., 4 Aug 2025).

1. Definition and task scope

ImageStoryGen-500K is presented as the main dataset used to test whether a large vision-LLM can turn an image plus a nuanced instruction into a long-form creative response that is both visually grounded and stylistically controlled (Jiang et al., 4 Aug 2025). The benchmark is designed around visually guided creative content generation, and the paper especially emphasizes story generation and poetry composition, since these tasks require both factual grounding in image content and nontrivial imagination.

The instruction side of the benchmark is explicitly more expressive than standard captioning prompts. Example instructions include “Write a poem about loneliness based on this image” and “Compose a short story describing the internal struggles of the character in the picture” (Jiang et al., 4 Aug 2025). From these examples, the benchmark’s instruction channel can specify genre, theme, tone, or rhetorical constraints. The provided description further indicates that the benchmark is intended to test long-form generation with abstract constraints such as melancholy, loneliness, hope, or internal struggle, which are not directly visible in the image but must be inferred and artistically expressed through grounding cues.

This makes ImageStoryGen-500K a benchmark for image-conditioned creative writing rather than for captioning, visual question answering, or generic multimodal instruction following. A plausible implication is that its difficulty lies not only in object recognition, but also in controlling discourse style while preserving fidelity to visual evidence.

2. Position within the VisuCraft framework

The dataset is used in VisuCraft as a downstream evaluation backbone rather than as a training source for the paper’s Multimodal Structured Information Extractor $\mathcal{E}$ (Jiang et al., 4 Aug 2025). The paper explicitly states that $\mathcal{E}$ is trained on image datasets like ImageNet, COCO, and OpenImages with rich annotations, while ImageStoryGen-500K is used for downstream benchmarking. This distinction is important because it positions the benchmark as an assessment resource for multimodal generation quality, not as a new pretraining corpus.

On each ImageStoryGen-500K sample, VisuCraft applies the pipeline

$T = \mathcal{M}(\mathcal{G}(\mathcal{E}(I), U))$

where $I$ is the image, $U$ is the user instruction, $\mathcal{E}$ extracts structured visual information, $\mathcal{G}$ converts that structure plus the instruction into an optimized prompt, and $\mathcal{M}$ is the underlying LVLM that produces the final text $\rightarrow$0 (Jiang et al., 4 Aug 2025).

The structured extractor is intended to output fine-grained descriptions rather than generic captions:

$\rightarrow$1

with $\rightarrow$2 described as a structured textual or JSON-like representation. The paper specifies that this output captures object poses, material properties, lighting conditions, spatial relationships, textures, color palettes, dominant light sources, emotional atmosphere, and implied narrative (Jiang et al., 4 Aug 2025). The prompt generation stage then combines this representation with the instruction,

$\rightarrow$3

and prioritizes task-relevant aspects such as atmosphere and emotion for poetry, or characters and interactions for story generation.

For ImageStoryGen-500K, this pipeline is motivated by the need to bridge visible evidence and abstract creative language. The paper gives the illustrative case that if an image suggests isolation, $\rightarrow$4 may extract details like “lone figure,” “desolate cliff,” or “flickering lighthouse beam,” and $\rightarrow$5 can transform those cues into a prompt that asks the backend LVLM to write a melancholic poem or story grounded in them (Jiang et al., 4 Aug 2025).

3. Supported tasks and benchmark difficulty

ImageStoryGen-500K supports multiple creative generation tasks. The paper especially highlights three:

1. Story generation — producing coherent narratives inspired by the image.
2. Poetry composition — producing metaphor-rich, emotionally aligned poems.
3. Advertising copy generation — more concise but still visually guided creative writing (Jiang et al., 4 Aug 2025).

The benchmark is used to test whether a model can understand visual details, infer latent mood or narrative, produce long-form text, obey style or tone constraints, and remain faithful rather than hallucinating (Jiang et al., 4 Aug 2025). This combination of requirements is presented as making the benchmark more challenging than standard captioning or VQA-style data.

Among the supported tasks, story generation and poetry are given the greatest emphasis. The reason stated in the source is that these genres require both factual grounding and imaginative transformation. Advertising copy generation is also included, but it appears less central in the discussion than the literary tasks.

The benchmark’s difficulty is tied to the fact that many of its prompts are expressive creative prompts rather than literal description prompts. This suggests that success depends on multimodal control under weakly explicit supervision: the model must infer which visual cues are relevant to a requested emotional or rhetorical mode, then sustain that mode over a long-form output.

4. Evaluation protocol and VisuGen Metrics

Evaluation on ImageStoryGen-500K uses the paper’s custom VisuGen Metrics: Visual Grounding (VG.), Creativity (C.), Instruction Adherence (IA.), and Mean (Jiang et al., 4 Aug 2025). The paper defines them conceptually as follows:

• VG. measures how accurately the output reflects image content.
• C. measures originality, imagination, and novelty.
• IA. measures how well the model follows the user instruction.
• Mean is the average of the three.

From the reported tables, the Mean is clearly computed as

$\rightarrow$6

The metric design reflects the benchmark’s central premise: visually guided creative generation cannot be reduced to relevance alone. A system may produce fluent text but fail grounding, or remain grounded but fail stylistic control, or obey instruction while becoming generic. The tripartite metric structure therefore decomposes performance into fidelity to image content, creative transformation, and adherence to the explicit prompt.

The benchmark also includes human evaluation. A human study over 200 samples per model, comprising 100 story samples and 100 poetry samples, uses a 5-point Likert scale and reports PVR, HCS, HIA, and OQ (Jiang et al., 4 Aug 2025). The source text states that these results reinforce the automatic metrics by showing that outputs are judged better by humans in relevance, creativity, adherence, and overall quality.

5. Quantitative results and ablation evidence

The main experimental table reports the following results on ImageStoryGen-500K (Jiang et al., 4 Aug 2025):

Model - Scenario	VG.	Mean
LVLM-Base – StoryGen	0.789	0.781
LVLM-Base – Poetry	0.771	0.772
LVLM-Enhanced – StoryGen	0.812	0.811
LVLM-Enhanced – Poetry	0.798	0.794
VisuCraft – StoryGen	0.825	0.822
VisuCraft – Poetry	0.810	0.810
VisuCraft – AdCopyGen	0.799	0.803

The key gains highlighted in the paper are that for story generation, VisuCraft improves over LVLM-Enhanced from 0.811 to 0.822 Mean and over LVLM-Base from 0.781 to 0.822; for poetry, VisuCraft reaches 0.810 Mean, versus 0.794 for LVLM-Enhanced and 0.772 for LVLM-Base (Jiang et al., 4 Aug 2025). The paper emphasizes that the biggest improvements are in Creativity and Instruction Adherence, which it interprets as consistent with the framework’s goal that stronger prompt structuring should make outputs more imaginative and more controllable.

Two ablation analyses are directly tied to the benchmark. The first removes either $\rightarrow$7 or $\rightarrow$8 on StoryGen:

Configuration	VG.	Mean
VisuCraft - w/o $\rightarrow$9	0.801	0.797
VisuCraft - w/o $\mathcal{E}$0	0.815	0.811
VisuCraft (Full)	0.825	0.822

The source text interprets these results as showing that $\mathcal{E}$1 is crucial for visual grounding, $\mathcal{E}$2 is crucial for turning extracted information into a strong generation prompt, and both are needed for the best score (Jiang et al., 4 Aug 2025).

The second ablation varies the granularity of the extractor output:

$\mathcal{E}$3 Output Granularity	VG.	Mean
Level 1: Basic Object Detection	0.775	0.769
Level 2: Objects + Attributes	0.803	0.800
Level 3: Full Structured Information	0.825	0.822

This result is particularly important for understanding the benchmark’s demands. The paper states that richer structured visual input materially improves creative output, and that just listing objects is not enough; attributes, relations, lighting, and atmosphere all matter (Jiang et al., 4 Aug 2025). In other words, ImageStoryGen-500K appears sensitive to representational granularity in the visual-conditioning interface.

Human evaluation results further support the same conclusion. On the 5-point Likert scale, VisuCraft outperforms the baselines in both story generation and poetry composition, with StoryGen scores of 4.25 PVR, 4.18 HCS, 4.30 HIA, and 4.24 OQ, and Poetry scores of 4.10, 4.05, 4.15, and 4.10 respectively (Jiang et al., 4 Aug 2025).

6. Dataset construction, documentation gaps, and likely biases

The paper describes ImageStoryGen-500K as “self-constructed,” but does not provide a detailed construction recipe beyond that characterization (Jiang et al., 4 Aug 2025). It also does not specify explicit train/validation/test splits, filtering thresholds, annotation workflows, inter-annotator agreement, deduplication procedures, or detailed quality-control pipelines. Those aspects are therefore undocumented in the provided paper text.

What is documented is that the dataset was curated to be diverse and difficult, with instructions that are not simple captioning prompts but expressive creative prompts, and that it was intended specifically for evaluating long-form creative image-to-text generation (Jiang et al., 4 Aug 2025). This suggests deliberate benchmark design around difficult image-plus-instruction pairings rather than around broad coverage of all multimodal generation settings.

Several likely constraints are identified in the source text, though not as a formal limitations section. Because the benchmark is self-constructed, it may reflect the authors’ curation choices and task preferences. The paper gives no evidence of broad multilingual coverage, so the benchmark appears English-centric. The task distribution seems centered on a few creative genres, especially story and poetry, which may bias the benchmark toward literary writing rather than other creative modalities. Since the dataset is designed for visually grounded creativity, it may favor images where mood or narrative is inferable, potentially underrepresenting images that are ambiguous or purely factual. Finally, because no detailed annotation protocol is provided, reproducibility and annotation consistency are unclear (Jiang et al., 4 Aug 2025).

Another plausible implication concerns methodological sensitivity. The benchmark seems to reward models that can exploit structured prompts well; thus, reported gains may partially reflect prompt-engineering sensitivity rather than only intrinsic generation ability. This does not invalidate the benchmark, but it affects how its results should be interpreted.

7. Relation to earlier image-conditioned storytelling work

Within the supplied research context, ImageStoryGen-500K stands in contrast to earlier, much simpler image-based story-generation pipelines such as “Multimodal Story Generation on Plural Images” (Jiang, 2020). That work defines story generation from multiple input images as generating a paragraph of story rather than separate captions, and uses a three-stage architecture consisting of an Image Captioner, a Relational Text Data Generator, and a Text Generator (Jiang, 2020). Its captioning stage uses Show, Attend and Tell with a pre-trained ResNet-101 encoder and an LSTM decoder with soft attention, while the final text generation stage uses GPT-2.

The conceptual connection is clear: both lines of work treat images as conditioning signals for longer-form textual generation rather than for single-sentence captioning. However, the differences are equally important. The earlier StoryGen work does not construct a 500K-scale image-story benchmark; it relies on Flickr8K, around 8,000 JPG images and around 40,000 captions, and procedurally creates a synthetic text corpus by extracting subjects, objects, and adjectives from captions and iteratively generating replacement-based prompts (Jiang, 2020). It reports no quantitative metrics, no human evaluation, and no benchmark design comparable to ImageStoryGen-500K.

The supplied description therefore positions StoryGen as an early conceptual precursor rather than a direct antecedent dataset release. Its relevance lies in the general idea that images can condition paragraph-level generation and that simple captioning is insufficient when textual outputs must relate multiple visual elements coherently. ImageStoryGen-500K extends that general problem setting into a large-scale benchmark for visually grounded creative generation with nuanced instructions, explicit metrics, and human evaluation (Jiang et al., 4 Aug 2025, Jiang, 2020).

Taken together, the evidence suggests that ImageStoryGen-500K functions as a stress test for weaknesses that standard LVLM prompting often exhibits in long-form creative tasks: weak grounding, generic outputs, and poor instruction adherence. In the VisuCraft paper, its central significance is precisely that it evaluates whether structured visual extraction and dynamic prompt construction improve performance on those dimensions (Jiang et al., 4 Aug 2025).