---
title: 'Openstory++: Instance-Aware Visual Storytelling'
url: https://www.emergentmind.com/topics/openstory-dataset
type: topic
---

# Openstory++: Instance-Aware Visual Storytelling

Openstory++ is a large-scale dataset and benchmark specifically designed for instance-aware, open-domain visual storytelling. It features comprehensive instance-level annotation—including consistent instance IDs, pixel-precise masks, and refined captions—across both single images and extended image sequences, addressing the challenge of maintaining instance coherence in multi-image generative and interpretive tasks. Openstory++ is accompanied by Cohere-Bench, a benchmark suite for rigorous quantitative and human evaluation of image generation tasks requiring long-horizon, multi-instance consistency [2408.03695].

## 1. Construction Pipeline

Openstory++ is created through an automated four-stage pipeline engineered to maximize annotation fidelity and temporal consistency:

1. **Keyframe Extraction and Deduplication**: I-frames are extracted from open-domain video corpora (primarily Pandas-70M and InternVid). DINOv2 encodes each frame, and near-duplicates (with high cosine similarity) are removed to ensure distinct visual appearance.
2. **Single-Image Captioning & Instance Masking**: Each keyframe receives an auto-generated caption from BLIP2. Entity nouns are extracted using NLTK, bounding boxes are detected with YOLO-Worldv2, and pixel-level instance masks are obtained via EfficientViT-SAM. Images with zero or more than eight instances are filtered out to control semantic complexity.
3. **Sequence-Level Alignment & Narrative Polishing**: Successive keyframes are grouped into short sequences. Video-LLaVA provides a coarse narrative, which is then refined by ChatGLM3-Turbo (a large language model) using both BLIP2 captions and Video-LLaVA storylines. The LLM ensures narrative cohesion and consistent instance naming across frames.
4. **Final Instance Masking & ID Consistency**: YOLO-Worldv2 is rerun to redetect entities in LLM-polished captions. Dinov2 features are fused with facial features to assign unique, persistent instance IDs, and EfficientViT-SAM generates the final pixel-level masks.

This multi-step process yields data suitable for training models to maintain narrative and entity coherence even in protracted open-domain visual stories.

## 2. Annotation and Quality Control

Each annotated image or frame in Openstory++ contains:

- Bounding boxes for every identified “main” entity (e.g., people, animals, vehicles)
- Pixel-accurate instance masks (using SAM), cropped per bounding box
- Unique, sequence-consistent instance IDs (e.g., “woman0,” “dog1”)
- Natural-language captions, LLM-refined to reference each instance by ID

The average number of instances per frame in sequence data is 2.5. Multiple filters are employed: 

- Aesthetic filter (unique frames: score >5; sequence frames: >4.5, as output by an aesthetic scoring model)
- Instance-count filter (1–8 instances)
- DINOv2 embedding-based deduplication
- LLM refinement for caption and naming quality
- Final manual sanity checks on a small held-out set

These measures collectively ensure granular, accurate, and semantically coherent annotation for both single frames and story sequences.

## 3. Dataset Scale and Characteristics

Openstory++ comprises two major subsets:

- **Single-Image Subset**: Approximately 100 million image–caption pairs, with each image annotated with 1–8 instances (mean 2–3), each instance masked and assigned an ID.
- **Sequence Subset**: One million short story sequences, averaging 28 frames per sequence, fully annotated frame-by-frame as above.

Frame resolution is standardized to 224 × 224 (for ViT) and 512 × 512 pixels (for Stable Diffusion conditioning). The captions, subject to LLM refinement, have an average length of 9–12 words and draw from a vocabulary of 50,000–100,000 unique tokens, supporting broad open-domain coverage.

## 4. Formal Definitions and Benchmark Metrics

Openstory++ defines the instance-aware visual storytelling task as generating $M$ images $I^1, ..., I^M$ from $M$ text prompts $L^1, ..., L^M$ in an interleaved, autoregressive manner:
$$
S_{txt} = \{L^1, ..., L^M\},\quad S_{img} = \{I^1, ..., I^M\}
$$

Cohere-Bench evaluates models using:

- **Pairwise cosine similarity** between feature embeddings:
  $$
  \text{Similarity} = \frac{1}{N} \sum_{i=1}^{N} (f^A_i \cdot f^B_i)^T
  $$
  where $f^A_i$, $f^B_i$ are normalized feature vectors.
- **Instance Integrity Score** using the Hungarian matching algorithm:
  $$
  \text{InstanceIntegrity} = \frac{1}{|F_{base}|} \sum_k \text{Similarity}(f^{current}_{i_k}, f^{base}_{j_k})
  $$
  with $(i_k, j_k)$ given by optimal matching on the cost matrix.

Additional evaluation metrics include:

| Metric                | Description                                      |
|-----------------------|--------------------------------------------------|
| Semantic Alignment    | CLIP cosine between generated image & text       |
| Background Consistency| CLIP similarity of backgrounds after inpainting  |
| Style Consistency     | DINOv2 cosine similarity across consecutive frames |
| Instance Consistency  | YOLO-World segment overlap + cosine (single/multi) |
| BLEU4                 | Caption similarity using BLIP2-generated captions|

## 5. Cohere-Bench: Task Design and Empirical Results

Cohere-Bench, provided alongside Openstory++, offers structured benchmarks for:

- **Story Generation**: Generate the first frame from text, then use the previous image and new text iteratively for subsequent frames.
- **Story Continuation**: Given image 1 and text 2, generate image 2, and so on.

Baseline models evaluated include DreamLLM, MiniGPT-5, SEED-X, Emu2, GPT4-V, MiniGemini, and Openstory++ reference models with and without visual annotations. Automatic metric scores highlight the leading performance of the Openstory++ model with visual annotations:

| Model                | Semantic | BG Consist. | Style | InstCons_s | InstCons_m | Integrity | BLEU4  |
|----------------------|----------|-------------|-------|------------|------------|-----------|--------|
| *Our model (vis ann.)* | 0.279    | 0.791       | 0.784 | 0.821      | 0.782      | 0.429     | 0.064  |
| GPT4-V               | close on semantic/style, lower on instance consistency |

Human evaluation covers criteria including text-image alignment, style consistency, story consistency, character consistency, plot continuity, image quality, and overall preference. Preference rates: Our model (21.28%) > GPT4-V (20.66%) > Emu2 (14.21%) > SEED-X (13.53%).

## 6. Comparative Analysis with Existing Datasets

Openstory++ is the only dataset to combine open-domain breadth, high per-instance annotation fidelity, and long temporally coherent sequences. The following table contrasts it with prior datasets:

| Dataset     | Domain | Caption   | Frames      | Avg. Length | Masks/Frame | Sequential |
|-------------|--------|-----------|-------------|-------------|-------------|------------|
| PororoSV    | Close  | Manual    | 73 K        | 5           | 0           | ✓          |
| Flintstones | Close  | Manual    | 123 K       | 5           | 0           | ✓          |
| DideMoSV    | Close  | Manual    | 53 K        | 3           | 0           | ✓          |
| VIST        | Open   | Manual    | 145 K       | 5           | 0           | ✓          |
| StorySalon  | Close  | ASR       | 159 K       | 14          | 1           | ✓          |
| SDD         | Open   | Generated | 76 M        | 1           | 3           | ✗          |
| Openstory++ | Open   | Generated | 100 M + 1 M seq | 28      | 2.5         | ✓          |

Openstory++ uniquely offers large scale, instance-level masks and IDs, and long sequences (mean length 28), filling both scale and granularity gaps present in previous resources.

## 7. Availability, Licensing, and Recommended Use

Openstory++ data and code are available at [https://openstorypp.github.io/]. The dataset is released under a CC BY 4.0 license; users must provide attribution and should confirm precise terms on the website. Recommended applications include training open-domain visual storytelling models requiring instance-awareness and temporal consistency, and evaluating MLLMs for entity consistency and narrative generation using Cohere-Bench.

Ethical use guidelines note that, despite substantial filtering, residual dataset biases (e.g., scene type and face blurring artifacts) may persist; downstream audits are advised. Attribution to the Openstory++ paper is requested for all uses [2408.03695].

Source: https://www.emergentmind.com/topics/openstory-dataset