FrameOracle-41K VideoQA Dataset
- The dataset provides minimal keyframes for VideoQA by annotating the exact frames needed to answer each question.
- It enables dual prediction training with both frame relevance ranking and count prediction for adaptive frame-budget models.
- Empirical results show modest accuracy gains and compute savings, underscoring its value for efficient video processing.
FrameOracle-41K is a VideoQA dataset introduced with the FrameOracle framework for adaptive video frame selection. It is described as the first large-scale VideoQA collection to provide keyframe annotations specifying the minimal set of frames required to answer each question, and it serves as the supervision source for the final supervised fine-tuning stage of FrameOracle’s dual prediction heads: one head predicts which frames are relevant to a query, and the other predicts how many frames are needed (Li et al., 4 Oct 2025).
1. Definition and role within FrameOracle
FrameOracle-41K is constructed to supervise a setting in which a vision-LLM must operate under a constrained frame budget while preserving answerability. In the underlying formulation, a video-question instance is associated not merely with an answer, but with a minimal subset of frames judged sufficient for answering that question. The dataset therefore differs from conventional VideoQA corpora that supply question-answer pairs without explicit supervision over temporal evidence selection.
Within FrameOracle, the dataset is used only in the final Stage 4 supervised fine-tuning step. The preceding stages rely on weaker proxy signals, including cross-modal similarity, whereas FrameOracle-41K provides direct supervision over both frame ranking and frame-count prediction. The paper states that this supervision refines the importance ranking head, stabilizes the -head’s count prediction, and yields consistent accuracy improvements and compute savings in downstream VideoQA tasks (Li et al., 4 Oct 2025).
A plausible implication is that the dataset is not merely an auxiliary benchmark artifact, but the central mechanism by which weakly guided frame selection is converted into explicitly supervised minimal-frame prediction.
2. Construction, source corpus, and dataset statistics
FrameOracle-41K is derived from LLaVA-Video-178K, a public VideoQA corpus of approximately video-question pairs spanning $2$–$3$ minute YouTube clips. From this pool, $64$ frames are uniformly sampled per clip, after which a two-stage pipeline is applied to discover the minimal subset of frames sufficient to answer each question. An instance is retained only if two conditions hold: an agent’s predicted answer matches the ground truth, and three independent VLMs all answer correctly using only the mined keyframes (Li et al., 4 Oct 2025).
The released dataset summary reports total video-question instances. All videos have durations between $2$ and $3$ minutes. The number of keyframes per instance has median approximately $5$ and mean approximately $7$; 0 of examples require at most 1 frames, while a small tail extends to approximately 2 frames. No official train/validation/test splits are released. In the reported experiments, all 3 instances are used for Stage 4 supervised fine-tuning, with approximately 4 held out for validation; users may re-split the corpus as desired.
| Property | Value |
|---|---|
| Source corpus | LLaVA-Video-178K |
| Total instances | 40,992 |
| Video duration | Between 2 and 3 minutes |
| Candidate frames per clip | 64 uniformly sampled frames |
| Keyframe count distribution | Median 5, mean 6 |
| Share needing 7 frames | 80% |
| Split policy | No official train/val/test split |
These statistics are consequential for the intended use case. The concentration of instances at relatively small keyframe counts indicates that many VideoQA questions in this corpus can be answered from sparse temporal evidence, while the long tail up to approximately 8 frames indicates that a fixed low budget would be insufficient for some questions. This directly matches the motivation for a learned count predictor rather than a fixed-budget selector.
3. Automated annotation protocol
FrameOracle-41K is collected entirely by automated annotators. The protocol explicitly states that no human labelers were used. Annotation proceeds in two stages: agent-based keyframe mining and filtering with verification (Li et al., 4 Oct 2025).
In Stage I, a Qwen2.5-VL-72B-based agent explores the 9 pre-sampled frames in a divide-and-conquer fashion. At each iteration, the agent examines $2$0 or $2$1 anchor frames, assigns each a relevance score from $2$2 to $2$3, and attempts to answer the question with a confidence. It then chooses which temporal segment to subdivide next based on summed relevance, continuing until the confidence is at least “high” or all frames have been visited. Any pair for which the agent’s final answer does not match the ground truth is discarded.
In Stage II, the frames visited by the agent are filtered by dropping those with relevance below a tunable threshold $2$4. The remaining keyframe set, together with the question, is then provided independently to three VLMs: Qwen2.5-VL-72B, LLaVA-OneVision-72B, and LLaVA-Video-72B. The keyframe set is accepted as sufficient only if all three models produce the correct answer. The agreement metric is therefore $2$5 model consensus on answer correctness. The paper further notes that very few instances are discarded at Stage II, which it presents as evidence of strong consistency.
This protocol addresses a common misconception that the dataset contains human-interpreted “important” frames in the ordinary annotation sense. It does not. Rather, the labels are machine-mined sufficiency sets validated by multi-model answer agreement. This suggests that the dataset operationalizes keyframes in terms of answer sufficiency under a specific automated verification regime, not in terms of generic saliency, semantics, or human attentional judgments.
4. Data schema and annotation semantics
Each FrameOracle-41K instance is stored as a single JSON object with a fixed schema. The fields are: id, video_path, duration, question, ground_truth_answer, num_selected_frames, keyframe_indices, and keyframes_dir (Li et al., 4 Oct 2025).
The id is a unique example identifier. video_path gives the path or URL to the original video. duration records the video length in seconds. question is the natural-language query, and ground_truth_answer is the target answer. num_selected_frames stores $2$6, the minimal number of frames. keyframe_indices is a sorted list of frame indices in $2$7, and keyframes_dir identifies the filesystem directory containing the extracted frames.
Two concrete examples are reported. One instance, with id: 30, concerns the question “What folding technique is demonstrated first in the video?” and has ground-truth answer “The 'SHIKAKU NO GI' (Square Fold) technique.” with num_selected_frames: 8 and keyframe_indices: [3, 7, 8, 12, 14, 17, 21, 25]. Another, with id: 1024, asks “Does the soccer player cross the line before kicking the ball?” and has ground-truth answer “Yes.” with num_selected_frames: 5 and keyframe_indices: [22, 23, 24, 26, 27].
The annotation semantics are narrow and technical. num_selected_frames and keyframe_indices encode the minimal sufficient evidence set discovered by the automated pipeline. A plausible implication is that these labels are tied to the $2$8-frame candidate sampling regime; they identify sufficiency relative to that discretized temporal support rather than to every possible frame in the underlying video stream.
5. Supervision signal and loss formulation
In Stage 4 supervised fine-tuning, FrameOracle-41K is used to train both prediction heads jointly. Let the candidate set size be $2$9 (for example, $3$0 or $3$1). The ground-truth keyframe indicator is $3$2 for each frame $3$3, and the ground-truth count is $3$4. The model outputs rank-head scores $3$5, normalized by $3$6, and a categorical $3$7-head distribution $3$8 for $3$9 (Li et al., 4 Oct 2025).
The rank head is trained as a per-frame binary classification problem using
$64$0
The $64$1-head is trained with cross-entropy against the one-hot ground-truth count:
$64$2
The total supervised loss is
$64$3
with $64$4 in practice.
This formulation makes the dataset structurally important in two ways. First, the per-frame mask $64$5 supervises ranking over candidate frames. Second, the scalar $64$6 supervises adaptive frame-budget prediction. In other words, the dataset does not merely indicate which frames matter; it also supplies direct supervision over how much visual evidence should be retained for a given query. That dual supervision is the defining technical role of FrameOracle-41K within the training pipeline.
6. Empirical impact, interpretation, and caveats
The paper’s ablation over training stages isolates the effect of Stage 4 supervision from FrameOracle-41K on Qwen2.5-VL-3B with $64$7. The random baseline reports $64$8 frames with average accuracy $64$9. Adding Stage 1 (SigLIP) gives 0 and 1. Adding Stage 2 (Rank LOO) gives 2 and 3. Adding Stage 3 (K head) gives 4 and 5. Adding Stage 4 (SFT) gives 6 and 7. In the paper’s summary of this comparison, Stage 4 reduces average frames from 8 to 9 ($2$0) and raises average accuracy by approximately $2$1 percentage points from $2$2 to $2$3 (Li et al., 4 Oct 2025).
End-to-end efficiency measurements are also reported on LLaVA-Video-7B. For a $2$4 setting, VLM FLOPs decrease from $2$5 to $2$6 T ($2$7), latency decreases from $2$8 to $2$9 s ($3$0), and average accuracy is maintained from $3$1 to $3$2. For a $3$3 setting, VLM FLOPs decrease from $3$4 to $3$5 T ($3$6), latency decreases from $3$7 to $3$8 s ($3$9), and accuracy increases by $5$0 percentage points from $5$1 to $5$2.
Several caveats follow directly from the dataset description. First, the annotations are entirely automated, so their validity depends on the behavior of the mining agent and the three verifying VLMs. Second, no official train/validation/test partition is released, which means comparisons across future uses may depend on user-defined re-splits. Third, the retained instances are filtered by answer correctness under both the agent and the $5$3 verifier consensus rule, so the corpus represents a subset of LLaVA-Video-178K for which this automated sufficiency criterion holds. These are not defects in themselves, but they define the precise scope of what FrameOracle-41K measures: minimal sufficient frame sets under a specific model-mediated protocol rather than human-annotated temporal evidence in a general sense.