ShotQA: Cinematic QA Dataset

• ShotQA is a large-scale multimodal question-answering dataset focused on professional cinematography, comprising ~70,000 QA pairs from over 243 feature films.
• It rigorously annotates eight cinematic dimensions—including shot size, framing, and camera movement—to benchmark vision-language model performance.
• The dataset employs automated shot extraction and expert-reviewed QA annotations to ensure high-quality, balanced data for robust cinematic analysis.

ShotQA is a large-scale, multimodal question-answering dataset created to advance vision-LLM (VLM) proficiency in professional cinematography at the single-shot level. The dataset is designed to teach and evaluate models on fine-grained film grammar across eight core cinematic dimensions using ∼70,000 multiple-choice QA pairs derived from over 58,000 high-quality still images and 1,200 video clips, predominantly curated from Oscar-nominated and Best Cinematography contender feature films released between 2000 and 2024. ShotQA is closely related to, and co-developed with, the ShotBench benchmark for expert-level cinematic language understanding (Liu et al., 26 Jun 2025).

1. Scope, Scale, and Data Provenance

ShotQA is the first dataset at this scale focused exclusively on professional cinematography at the single-shot level. It comprises approximately 70,000 QA pairs sampled from 243 distinct feature films spanning multiple genres, ensuring coverage of stylistic diversity in mainstream and auteur cinema. Images are selected for both technical and aesthetic quality using automated filtering via the LAION aesthetic predictor; full-length video sources are segmented into shots with TransNetV2, and video frames are post-processed using FFmpeg to remove letterboxing.

Each QA pair is associated with the film title, shot ID, frame number or timestamp, and a specific cinematography dimension, supporting detailed modeling and fine-grained analysis.

2. Cinematography Dimensions and Question Taxonomy

ShotQA addresses eight core cinematographic dimensions, each rooted in professional standards and industry terminology as found in sources such as Studiobinder guides. The question taxonomy maps each QA example to one targeted aspect:

Dimension	Example Categories	Approximate QA Pairs
Shot Size (SS)	Extreme Close-Up, Close-Up, Medium, Wide	8,579
Shot Framing (SF)	Single, Two-Shot, Over-the-Shoulder, Group	8,298
Camera Angle (CA)	Low Angle, High Angle, Dutch Angle, Aerial	9,405
Lens Size (LS)	Ultra-Wide/Fisheye, Wide, Medium, Long	8,324
Lighting Type (LT)	Daylight, Artificial, Mixed, Firelight	6,811
Lighting Condition (LC)	Soft Light, Hard Light, Backlight, Silhouette	8,778
Composition (SC)	Centered, Left-Heavy, Right-Heavy, Short-Side	9,597
Camera Movement (CM)	Static, Push-In, Pull-Out, Pan, Zoom, Boom	1,200

Dimension definitions and decision cues are derived from cinematic authorities; for instance, a “Medium Close-Up” is defined as a subject framed from the chest to forehead (focal lengths ∼50–85 mm), while “Pull-Out” signifies physical camera movement backward, resulting in increased parallax and background scale.

3. Data Annotation and Quality Assurance

Shot extraction for still images is filtered for aesthetic quality and NSFW content. For videos, automated shot boundary detection is performed prior to frame cropping. Human annotators are trained via curated tutorials for all eight dimensions, followed by pilot annotation rounds audited by domain experts to establish guideline consistency.

The QA annotation protocol involves:

• Auto-generating questions from templates.
• Pre-populating candidate labels using metadata from Shotdeck, a professional cinematography database.
• For video camera movement questions, annotators mark gesture start/end times and verify correctness.
• Multi-stage expert review audits with guidelines updated iteratively.

Quality is monitored via inter-annotator agreement, with low-agreement samples re-annotated or excluded. The original high-quality subset of 3,572 QA pairs—spanning 3,049 images and 464 clips—forms the basis for ShotBench verification before scaling annotation to the final ∼70,000 pairs with standards maintained.

4. Dataset Structure, Formats, and Distribution

ShotQA is distributed as a series of JSON files, organized into train (≈56,000), validation (≈7,000), and test (≈7,000) splits. Each entry contains:

• shot_id: unique identifier
• film_title: string
• source_type: “image” or “video”
• timestamp_or_frame: integer (frame index or timestamp)
• dimension: one of {SS, SF, CA, LS, LT, LC, SC, CM}
• question_text: string
• choices: string array (usually 4–5)
• answer: string (correct choice)
• Optional: focal_length, aspect_ratio, resolution

Split proportions are approximately 80%/10%/10% for train/val/test. Frame resolution is capped at 360 × 640 px for evaluation, with original assets up to 1920 × 1080 px. Aspect ratios are predominantly 16:9 (∼70%) and 2.39:1 (∼30%). Per-dimension class balance is maintained within ±5% of the mean, except for rare camera moves (e.g., Dolly-Zoom ≈0.6%).

Example test set entries illustrate the format and complexity:

$1(\cdot)$0 $1(\cdot)$1

5. Evaluation Methodology and Metrics

The primary evaluation metric for models trained or tested on ShotQA is top-1 accuracy:

$$\Delta \mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} 1(\hat{y}_i = y_i)$$

where $1(\cdot)$ is the indicator function, $N$ is the number of QA examples, and $\hat{y}_i$, $y_i$ are the predicted and ground-truth labels, respectively. When relevant, secondary metrics (precision, recall, F1-score) are reported per category for detailed analysis. The training regime leverages supervised fine-tuning (SFT) using cross-entropy loss ($L_{CE}$) and, for advanced models, Group Relative Policy Optimization (GRPO), whose objective is:

$$\mathcal{L}_{GRPO} = \frac{1}{G}\sum_{i=1}^{G} \min\left( \frac{\pi_\theta(o_i|x)}{\pi_{\theta_{\text{old}}}(o_i|x)}A_i,\; \mathrm{clip}\bigg(\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}},\,1-\epsilon,\,1+\epsilon\bigg)A_i \right)$$

where $\pi_\theta$ is the updated policy, $A_i$ the advantage, and $\epsilon$ the clipping parameter.

ShotQA has enabled the development of the ShotVL model, which, when trained with this dataset using the above objectives, establishes a new performance standard on the ShotBench evaluation set (Liu et al., 26 Jun 2025).

6. Coverage, Sampling, and Class Distribution

ShotQA is constructed to ensure representative sampling across films, genres, and cinematic grammar dimensions. Stratified sampling is employed to maximize coverage both within each class (dimension) and across the corpus of 243 films. Most categories are balanced within ±5% of the mean sample count per class, mitigating bias and enabling robust error analysis. Camera Movement is a notable exception, with rarer types (such as Dolly-Zoom) being inherently less frequent by occurrence in the source material (~0.6%).

This construction supports precise benchmarking of VLMs for nuanced visual reasoning, spatial relationships, and fine-grained recognition of established film conventions.

7. Intended Applications, Accessibility, and Licensing

ShotQA is designed for:

• Fine-tuning vision-LLMs to enhance cinematography-aware image and video understanding and generation.
• Benchmarking model performance in the domain of professional film grammar.
• Supporting AI-assisted shot planning, cinematic style matching, and automated editing suggestions in production workflows.

The dataset, along with associated models and code, is openly released under an MIT license. Download links, detailed documentation, and accompanying resources are provided at https://vchitect.github.io/ShotBench-project/.

ShotQA’s scale, expert-based annotation, comprehensive coverage, and rigorously balanced taxonomy position it as a critical resource for film-aware VLM research and professional tool development (Liu et al., 26 Jun 2025).