RefineShot: Cinematography Benchmark

• RefineShot is a benchmark for cinematography understanding that reworks ShotBench to eliminate ambiguous option design and unreliable reasoning.
• It formalizes question options by sampling distractors from the same descriptive dimension, ensuring consistent granularity and improved evaluation reliability.
• The framework introduces reliability metrics like Faithful Reasoning Score and Instruction Adherence Score to assess both answer accuracy and reasoning consistency.

Searching arXiv for the RefineShot benchmark and closely related cinematography-understanding work. Searching arXiv for “RefineShot cinematography understanding ShotBench ShotVL”. Searching arXiv now. {"query":"RefineShot cinematography understanding ShotBench ShotVL", "max_results": 10} to=arxiv_search 全民彩票天天送"query":"RefineShot cinematography understanding ShotBench ShotVL","max_results":10} RefineShot is a refined and expanded benchmark for cinematography understanding that reworks ShotBench and redefines its evaluation protocol in order to make comparisons among vision-LLMs more reliable (Wu et al., 2 Oct 2025). In this setting, cinematography understanding denotes the ability to recognize not only the visual content of a scene but also the cinematic techniques that shape narrative meaning, including camera movement, framing, lighting, and composition. RefineShot is motivated by two specific defects identified in prior evaluation practice: ambiguous option design at the dataset level and unreliable reasoning behavior at the model level. Its central contribution is therefore not a new model architecture, but a stricter benchmark-and-protocol framework that jointly measures task accuracy, reasoning faithfulness, and instruction adherence (Wu et al., 2 Oct 2025).

1. Origin in cinematography benchmark design

RefineShot arises from a re-examination of ShotBench and its state-of-the-art baseline ShotVL (Wu et al., 2 Oct 2025). ShotBench is described as the first large-scale, expert-annotated multiple-choice benchmark for cinematography understanding, covering eight cinematic dimensions: lens source, lighting type, lighting condition, shot framing, shot size, camera angle, shot composition, and camera movement. ShotVL achieved top accuracy on that benchmark, but RefineShot argues that the apparent progress was undermined by defects in both the benchmark construction and the interpretation of model scores (Wu et al., 2 Oct 2025).

The first defect is ambiguous option design. In the original ShotBench formulation, a single question could mix descriptors drawn from different descriptive dimensions. The paper highlights lighting condition as a representative case, where illumination direction labels such as “side light” and “backlight” were mixed with contrast descriptors such as “high contrast” and intensity descriptors such as “soft light.” Because such options are not mutually exclusive and do not reside at a uniform level of granularity, multiple answers can become defensible within the same item (Wu et al., 2 Oct 2025).

The second defect is unreliable reasoning. RefineShot reports that ShotVL’s high answer accuracy did not imply reliable internal consistency. Controlled experiments indicated both reasoning unfaithfulness, in which correct answers contradicted the model’s own chain-of-thought, and poor instruction adherence, in which performance degraded sharply under simple step-by-step instructions (Wu et al., 2 Oct 2025). RefineShot therefore reframes the evaluation problem: the benchmark must assess not only whether a model selects the right option, but also whether its reasoning supports that option and whether it follows the requested response format.

2. Formal reconstruction of question options

RefineShot formalizes a ShotBench item as a question $q$ with option set

$$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$

where each option $o \in O_q$ is mapped to a descriptive dimension $d$ by $M(o)=d$ (Wu et al., 2 Oct 2025). The benchmark critique is expressed through the condition

$\exists\, o_i, o_j \in O_q:\; M(o_i)\neq M(o_j),$

which indicates that the original option set mixed incompatible dimensions and thereby violated mutual exclusivity (Wu et al., 2 Oct 2025).

To enforce consistency, RefineShot replaces each original option set $O_q$ with a refined set

$$O'_q = \{a_q\}\,\cup\,\mathrm{Sample}\bigl(S_{M(a_q)}\setminus\{a_q\},\,k-1\bigr),$$

where $S_{M(a_q)}$ is the set of all descriptors in the same dimension as the ground-truth answer $a_q$ (Wu et al., 2 Oct 2025). The distractors are therefore sampled from the same semantic axis as the answer, after which the options are shuffled to remove ordering bias. In the rare case that $$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$0, the multi-choice question is converted into a binary judgment

$$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$1

asking whether the image exhibits that property (Wu et al., 2 Oct 2025).

Applying this procedure across all eight tasks, RefineShot revises 961 questions (Wu et al., 2 Oct 2025). The stated goal is a benchmark in which every question’s options share the same dimension and the same granularity. A representative example is the treatment of lighting-condition labels: categories such as “Firelight,” described in the paper as a functional category, are mapped into a purely physical taxonomy including “LED,” “HMI,” “Sunny,” and “Overcast.” Under these consistent, mutually exclusive options, “Overcast” recognition accuracy rises from roughly 75% in the original benchmark to 97.3% (Wu et al., 2 Oct 2025).

3. Reliability metrics beyond raw accuracy

RefineShot extends evaluation from a single scalar accuracy score to a metric tuple

$$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$2

intended to capture not only what answer a model gives, but also how faithfully and how compliantly it produces that answer (Wu et al., 2 Oct 2025).

The Faithful Reasoning Score (FRS) is defined over $$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$3 evaluation samples. For sample $$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$4, the indicator

$$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$5

records whether the chain-of-thought supports the final response. The score is then

$$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$6

This metric operationalizes reasoning faithfulness as entailment between the model’s own intermediate rationale and its final prediction (Wu et al., 2 Oct 2025).

The Instruction Adherence Score (IAS) is defined from two accuracies: $$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$7, the standard accuracy under direct prompting, and $$O_q = \{o_{q,1}, o_{q,2}, \dots, o_{q,k}\}, \quad a_q \in O_q,$$8, the accuracy obtained when correctness and format compliance are both enforced (Wu et al., 2 Oct 2025). IAS is the ratio of the adhered accuracy to the original accuracy. In effect, it measures how much of a model’s nominal performance survives once the model is required to follow a step-by-step reasoning format with proper tags and enumerated steps.

This protocol changes the semantics of benchmark evaluation. Under RefineShot, a model can no longer be treated as strong merely because it answers many questions correctly under permissive prompting. High performance must persist under reasoning checks and format constraints. This suggests that RefineShot is designed as a diagnostic framework rather than a pure leaderboard instrument (Wu et al., 2 Oct 2025).

4. Evaluation harness and protocol modularity

RefineShot’s algorithmic novelty lies in the benchmark construction and evaluation wrappers rather than in modifications to model architecture (Wu et al., 2 Oct 2025). The paper describes an Automated Verifier that uses Qwen-3B as a black-box verifier for both FRS and IAS. In this setup, Qwen-3B evaluates whether a chain-of-thought is consistent with the final answer and whether the response satisfies the specified reasoning format (Wu et al., 2 Oct 2025).

The benchmark also introduces Modular Protocols. The three protocols—direct accuracy, a reasoning-check protocol, and a step-by-step adherence protocol—are implemented as plug-in modules that can be toggled on any vision-LLM under test without changing model weights or internal structure (Wu et al., 2 Oct 2025). The paper is explicit that no changes to ShotVL or Qwen internals are required. RefineShot therefore functions as an external harness that exposes defects hidden by simpler evaluation procedures.

This design has methodological significance. Because the evaluation logic is factored outside the model, the benchmark isolates dataset quality and response reliability from architectural differences. A plausible implication is that RefineShot aims to make model comparisons less sensitive to idiosyncratic prompting and more sensitive to genuine competence in structured visual reasoning.

5. Empirical profile and diagnostic findings

On the refined benchmark, RefineShot reports the following overall accuracies and reliability scores: Qwen2.5VL-3B reaches 47.5 overall accuracy, 98.5 FRS, and 87.8 IAS; Qwen2.5VL-7B reaches 51.7 overall accuracy, 98.9 FRS, and 93.5 IAS; ShotVL-3B reaches 67.8 overall accuracy, 83.2 FRS, and 16.4 IAS; ShotVL-7B reaches 70.2 overall accuracy, 93.0 FRS, and 11.7 IAS (Wu et al., 2 Oct 2025).

These results reveal a pronounced accuracy–reliability gap. ShotVL-7B remains the strongest model in raw accuracy at 70.2%, yet its IAS is only 11.7%, meaning that under the enforced step-by-step format its performance is largely not preserved (Wu et al., 2 Oct 2025). By contrast, Qwen2.5VL-7B has lower raw accuracy but a much higher IAS of 93.5%, indicating that its correct behavior is substantially more compatible with explicit instruction following.

The reasoning-check protocol adds a second diagnostic layer. RefineShot reports that ShotVL-3B’s apparent 68.3% accuracy drops to 59.0% when only faithfully reasoned outputs are counted, a decrease of 8.9 points, whereas Qwen models remain stable with at most a 0.8-point drop (Wu et al., 2 Oct 2025). The benchmark’s claim is therefore not simply that some models are weaker than others, but that raw multiple-choice accuracy can conceal substantial inconsistencies between answer generation, reasoning traces, and prompt compliance.

A common misconception addressed by these findings is that benchmark leadership in answer accuracy necessarily reflects robust cinematography understanding. RefineShot rejects that equivalence. Under its protocol, accuracy is only one component of competence; reasoning faithfulness and adherence to explicit inference structure are treated as co-equal evaluation targets (Wu et al., 2 Oct 2025).

6. Implications for multimodal reasoning in film and media

RefineShot situates cinematography understanding as a capability relevant to multimodal understanding in real-world applications and to coherent content creation in film and media (Wu et al., 2 Oct 2025). Within that framing, the benchmark’s contribution is to raise evaluation standards for models intended to reason about cinematic techniques rather than merely classify them.

The paper identifies three broader implications. First, in benchmark rigor, it argues that eliminating heterogeneous distractors and measuring reasoning faithfulness should become standard practice beyond cinematography-specific tasks (Wu et al., 2 Oct 2025). Second, in model improvement targets, low IAS and FRS scores indicate that future multimodal LLMs must not only learn visual taxonomies but also internalize and execute structured reasoning protocols under instruction (Wu et al., 2 Oct 2025). Third, in creative applications, domains such as film education, automated shot logging, and intelligent cinematography assistants require not just raw answer accuracy but also reliable reasoning and compliance with creative prompts (Wu et al., 2 Oct 2025).

RefineShot therefore redefines what counts as progress in cinematic understanding. The benchmark does not claim that models should only predict cinematic labels more accurately; it requires that they do so under a question design free of dimension mixing, with chain-of-thought that supports the answer, and with adherence to explicitly requested reasoning structure. In that sense, RefineShot converts ShotBench from a purely evaluative benchmark into a diagnostic framework for film-style visual reasoning (Wu et al., 2 Oct 2025).