---
title: 'SpecTemp-80K Dataset: Dual-Level Video QA'
url: https://www.emergentmind.com/topics/spectemp-80k-dataset
type: topic
---

# SpecTemp-80K Dataset: Dual-Level Video QA

SpecTemp-80K is a large-scale, dual-level annotated dataset for speculative temporal reasoning in long video understanding. Designed to support the training and evaluation of cooperative dual-model frameworks, notably the SpecTemp architecture, the dataset contains 80,142 video–question–answer (QA) triplets spanning a broad temporal spectrum and nine distinct video corpora. Each sample provides synchronized supervision at both coarse (segment-level) and fine (frame-level) temporal resolutions, enabling efficient and accurate temporal evidence localization and reasoning for video multi-modal large language models (MLLMs) [2512.00805].

## 1. Construction and Scope

SpecTemp-80K was constructed to address the limitations of the “thinking-with-frames” paradigm in long video reasoning by providing rich, multi-granular supervision for RL and other learning paradigms that require both segment- and frame-level information. The dataset aggregates N=80,142 video–QA triplets from nine video sources across three temporal categories:

- **Short-form (<1 min):** CLEVRER, PerceptionTest, STAR, NeXT-GQA (32.4% of videos)
- **Medium-length (1–10 min):** LLaVA-Video, ActivityNet, YouCook2 (51.8% of videos)
- **Long-form (>10 min):** MovieChat, Ego4D (15.8% of videos)

The diversity of source domains and temporal regimes (sub-minute to tens of minutes) ensures coverage of a wide range of real-world video understanding scenarios. Precise total dataset duration in hours is not reported, but the proportional splits emphasize robust representation of both brief and extended contexts.

## 2. Annotation Protocol

Each QA sample is annotated at two synchronized levels:

**(a) Coarse Evidence Spans ($S_i$):**  
Contiguous temporal segments relevant to answering the query, denoted
$$
S_i = \{ (t_\text{start}^j, t_\text{end}^j) \mid 0 \leq t_\text{start}^j < t_\text{end}^j \leq d_i, \; j=1\dots M_i \}
$$
where $d_i$ is video duration. Each $(t_\text{start}, t_\text{end})$ corresponds to a “<segment>” token in the model's reasoning trajectory.

**(b) Fine-Grained Frame-Level Evidence ($F_i$):**  
Within each coarse span, a subset of representative frames is selected as most informative:
$$
F_i = \{ f_k \mid f_k \in \mathbb{N}, \; t(f_k) \in \bigcup_j [t_\text{start}^j, t_\text{end}^j], \; k=1\dots K_i \}
$$
with $t(f_k)$ as frame timestamp. These are surfaced as “<frame>” tokens.

**Annotation Procedure:**
- **Preliminary Annotations** via GPT-4o generate scene- and frame-level captions for events and visual cues.
- **Trajectory Generation**: GPT-4o simulates multi-round speculation–verification traces marked by <think>, <segment>, <frame>, and <answer>.
- **Manual Validation**: A random subset is manually checked for temporal and frame correctness; errors are culled.

**Empirical Statistics:**
- Average $\langle M_i \rangle \approx 1.8$ spans per video (each question typically triggers one initial and up to 2 additional segment predictions)
- Median span length ≈ 15 seconds (long-tailed toward much longer spans)
- Average $\langle K_i \rangle \approx 4.2$ frames per video (draft model selects 2 frames per iteration, up to 3 iterations)

## 3. Data Format and Mathematical Schema

SpecTemp-80K's organization supports streamlined loading and evaluation:

**Directory Structure:**
- `train/`, `val/`, `test/` (80%/10%/10% splits)
    - `videos/`: video files or links
    - `annotations.jsonl`: one sample per line

**JSONL Schema:**
```json
{
  "video_id": "string",
  "duration": float,
  "question": "string",
  "answer": "string",
  "spans": [ {"start": float, "end": float}, ... ],
  "frames": [ int, ... ],
  "trajectory": [
    {
      "round": int,
      "think": "string",
      "segment": [start, end] or null,
      "draft_frames": [f1, f2]
    }, ...
  ]
}
```

**Mathematical Formalization:**
- $V = \{ v_1, \dots, v_N \}$, $N=80\,142$
- For each $v_i$:
    - $d_i \in \mathbb{R}^+$
    - $S_i$ and $F_i$ as above
    - Full dataset: $A = \{ (v_i, q_i, a_i, S_i, F_i, \text{Traj}_i) \}_{i=1\dots N}$

## 4. Splits, Annotation Statistics, and Distributions

The dataset is partitioned as follows:

| Split  | Samples | ⟨#Spans⟩ | ⟨#Frames⟩ |
|--------|---------|----------|-----------|
| Train  | 64,113  | 1.79     | 4.25      |
| Val    | 8,014   | 1.82     | 4.18      |
| Test   | 8,015   | 1.81     | 4.22      |

**Segment Length and Frame Density:**
- Coarse-span length $L$ has a right-skewed distribution $P(L)$: ~50% of spans <12 seconds, 10% >60 seconds.
- Within spans, frame density ($K_i/d_i$) ≈ 0.1 frames/sec, consistent with 2 frames per ~20 seconds.

This suggests the annotation process closely mirrors real-world sparsity in answer-relevant evidence, supporting efficient learning of temporal localization.

## 5. Applications and Benchmarking

**Recommended Use Cases:**
- **Temporal Evidence Retrieval:** Training models to localize relevant segments.
- **Frame-Level Summarization:** Learning to select key frames that condense essential information.
- **Long-Video Question Answering:** Joint training with coarse and fine temporal cues.
- **Reinforcement Learning for Frame Sampling:** Policy optimization using dual-level annotated supervision.

**Benchmark Results:**
- Training SpecTemp on SpecTemp-80K achieves up to a +12% absolute gain (on Video-Holmes benchmark), and 19–23% faster inference compared to prior “thinking-with-frames” paradigms.
- IoU for segment localization increases by 3–5%; frame processing latency drops correspondingly.

A plausible implication is that models leveraging SpecTemp-80K’s structured dual supervision are intrinsically better at balancing accuracy and efficiency in QA over long video contexts.

**Integration Strategies:**
- Pre-train or fine-tune multi-modal LLMs using synchronized span/frame labels.
- Use IoU and information gain measured on $S_i$ and $F_i$ for evaluating novel selection policies.
- Adopt standardized splits for reporting accuracy, localization IoU, and frame selection precision.

## 6. Significance and Impact

SpecTemp-80K provides a critical resource for advancing research on efficient, scalable, speculation-driven video understanding. Its explicit, dual-level annotations and balanced coverage across short, medium, and long video regimes make it well-suited for developing RL and LLM-based approaches that require both global temporal coherence and fine-grained perceptual grounding. The dataset’s format and rigorously curated splits facilitate reproducibility and fair benchmarking across the video QA and temporal evidence retrieval research communities [2512.00805].

Source: https://www.emergentmind.com/topics/spectemp-80k-dataset