---
title: 'JumpScore: Fine-Grained Temporal Benchmark'
url: https://www.emergentmind.com/topics/jumpscore-benchmark
type: topic
---

# JumpScore: Fine-Grained Temporal Benchmark

JumpScore is a temporal localization benchmark designed to measure perceptual transition-level grounding in high-frequency, densely repeated motions, with the primary case study being jump-rope cycles. Diverging from standard temporal-grounding evaluations focused on a handful of discrete, separated events, JumpScore targets the precise localization of every repeat event—where events are visually similar and separated by as little as 0.4 s—in long, naturalistic video clips. The benchmark’s formulation, dataset curation, evaluation protocol, and integration with codec-stream tokenization address a critical underrepresented regime in video understanding: event-level detection in repetitive, transition-driven motion [2605.25979].

## 1. Motivation and Benchmark Scope

JumpScore was motivated by the observation that existing temporal grounding and video QA benchmarks (e.g., Charades-STA, ActivityNet Captions) are poorly aligned with settings requiring fine-grained, repeated event detection. In jump-rope videos, consecutive cycles are tightly packed (often 0.4 s apart) and visually indistinguishable except for a brief, salient “transition” moment: the instant when the rope passes behind the jumper’s legs. Standard approaches relying on uniform frame sampling or event deduplication struggle to resolve such instances, leading to substantial precision and recall degradation. JumpScore directly isolates this “perceptual transition” regime, serving both as a testbed for dense temporal grounding and as a stress test for token allocation strategies in video-language models [2605.25979].

## 2. Task Formulation and Metric

The core task of JumpScore is to produce a ranked list of timestamps corresponding to every cycle start in a given long-form jump-rope video. The input is a 30–90 s clip, typically at resolutions of 720p or higher. The fixed prompt for every video establishes the event definition: “List the start timestamps in s of each jump-rope the main character does in the video. The start is defined as the moment the rope is behind the legs.” Outputs are decimal-second timestamp lists.

JumpScore is scored by mean Average Precision (mAP) averaged over three temporal tolerances (\(\delta = 0.1, 0.2, 0.3\) s). Each predicted timestamp \(t_p\) is a true positive if it falls within \(\delta\) s of an unmatched ground-truth cycle start \(t_g\); otherwise, it is a false positive. Unmatched ground-truths are counted as false negatives. Varying confidence thresholds or list length yields precision-recall curves and AP at each tolerance, and the final score is the mean of AP@0.1, AP@0.2, and AP@0.3 s:
\[
\mathrm{JumpScore} = \frac{1}{3}\left[ \mathrm{AP}(0.1) + \mathrm{AP}(0.2) + \mathrm{AP}(0.3) \right].
\]
The 0.1 s regime corresponds to a half-cycle tolerance, while 0.3 s is the most permissive.

## 3. Dataset Construction and Annotation

The JumpScore dataset comprises 189 in-the-wild jump-rope videos, each between 30 and 90 seconds in length. All videos are at least 720p, with 84% reaching 1080p, ensuring sufficient fidelity to unambiguously resolve the “rope-behind-legs” visual cue. Every frame is decoded at its native frame rate, with human annotators marking each cycle start (the precise disappearance of the rope behind the legs) to decimal-second precision. Subsequent cross-checking aligns each annotation to decoded frame times, enforcing direct correspondence. Clips typically contain tens to hundreds of cycles, thus the benchmark requires both high recall (no cycle omitted) and boundary-precision (localizing onset within a narrow window) [2605.25979].

## 4. Evaluation Protocol

The evaluation protocol in JumpScore consists of the following steps:

1. The fixed prompt is issued to the model, which outputs a ranked list of event timestamps.
2. For each specified tolerance \(\delta\), a predicted timestamp is greedily matched to the nearest unmatched ground-truth event within \(\delta\) s. Any unpaired prediction is a false positive; any unpaired ground-truth, a false negative.
3. Precision–recall curves are computed for varying list lengths; AP is calculated for each \(\delta\).
4. The JumpScore is reported as the mean of the three APs.

This protocol stresses event-level completeness and localization precision, given the density and subtlety of events in the underlying data.

## 5. Codec-Stream Tokenization and its Role

JumpScore’s construction was directly influenced by the limitations of uniform frame sampling for sub-second, repeat-event localization. LLaVA-OneVision-2 introduced codec-stream tokenization, a video representation method observing the compressed bitstream directly, instead of selecting frames at uniform intervals. This approach adaptively allocates tokenization to high-bit-cost (i.e., high-motion or high-residual) regions of the video, yielding temporal groups that contract around rapid motion and expand during predictable spans.

Formally, the adaptive grouping procedure divides the video into $B$ fixed-duration bins, computes per-bin codec bit-costs \(e_b\), derives a quota $\theta$ for per-group bit allocation, then greedily extends each temporal group until the quota is reached and a suitable low-motion “valley” is found for group boundarying. Codec spatial scoring further selects high-saliency patches within each group based on percentile-normalized motion vectors and luma residuals, aggregating responses to guide compact canvas construction.

JumpScore’s ground-truth cycle onsets coincide with peaks in these motion/residual signals, so codec-stream tokenization aligns representation capacity precisely with event transitions. Under any constrained token budget, this leads to significant accuracy gains compared to uniform sampling, which frequently fails to capture the event-aligned frames.

## 6. Quantitative Results, Budget Sensitivity, and Ablations

Empirical results underline the benchmark’s challenge and discriminatory power. The 8B LLaVA-OneVision-2 model achieves 74.9 JumpScore mAP, a 44.8-point improvement over the Qwen3-VL-8B baseline (30.1). Other multimodal models (Keye-VL-1.5: 39.6; InternVL-3.5: 11.0; PLM-8B: 13.1; LLaVA-OV-1.5: 2.1) show much lower performance.

Under matched visual-token budgets, codec-stream tokenization outperforms uniform sampling by +17.3 mAP on JumpScore. The benefit is also confirmed across a range of token budgets:

| Frame Budget | Uniform mAP | Codec-stream mAP |
|--------------|-------------|------------------|
| 4            | 32.5        | 39.4             |
| 8            | 35.2        | 40.2             |
| 16           | 36.7        | 46.9             |
| 32           | 37.6        | 58.3             |
| 64           | 39.9        | 71.3             |
| 128          | 45.4        | 74.9             |

At 32–128 tokens, codec-stream yields rapid mAP increases from ~58 to ~75, while uniform sampling fails to exceed ~46. At low budgets (4–16 frames), uniform sampling often misses the cycle-onset frames. Smaller model variants (e.g., LLaVA-OV-2-1.5B) remain ineffective (≈2 mAP), indicating joint necessity for capacity and aligned tokenization. These findings demonstrate JumpScore’s sensitivity to both token-allocation strategy and model scale [2605.25979].

## 7. Context, Significance, and Related Evaluation Paradigms

JumpScore fills a vital benchmark gap by demanding event-level precision in settings with numerous, rapidly repeating, and visually ambiguous events. The benchmark exposes a key weakness of approaches rooted in uniform or similarity-based frame selection, especially for long-video temporal localization and VLMs. Its construction and results suggest that for next-generation perceptual intelligence, token allocation should be driven by signal structure—bit-cost and residual—rather than global frame-rate or elapsed time.

Related metrics include spatial grounding mAP (where predicted regions are compared to ground-truth boxes across IoU thresholds) and region-overlap/boundary F-measure ($\mathcal{J}\mathcal{F}$) in video object segmentation. However, these do not address the high-frequency transition-localization regime that JumpScore uniquely targets.

Taken together, JumpScore establishes a rigorous standard for fine-grained, repetitive event detection and provides a stringent testbed for models leveraging codec-aligned perceptual representations [2605.25979].

Source: https://www.emergentmind.com/topics/jumpscore-benchmark