PushupBench: Your VLM is not good at counting pushups

Abstract: Large vision-LLMs (VLMs) can recognize \textit{what} happens in video but fail to count \textit{how many} times. We introduce \textbf{PushupBench}, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1\% exact accuracy; open-source 4B models score $\sim$6\%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in \texttt{lmms-eval} (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)

• The paper introduces PushupBench as a novel benchmark highlighting VLMs' inability to accurately count repetitive actions in long-form fitness videos.
• It employs detailed annotation and robust metrics—exact match, MAE, and R²—to expose mode collapse and reward hacking phenomena in counting tasks.
• Counting-fine-tuning via reinforcement learning not only improves repetition counting but also enhances broader temporal reasoning in vision-language models.

PushupBench: Evaluating the Counting Proficiency of Vision-LLMs in Repetitive Action Video

Introduction and Motivation

PushupBench addresses the critical failure of current large Vision-LLMs (VLMs) in fine-grained temporal reasoning, particularly counting repetitive actions in videos. While VLMs exhibit strong performance on standard semantic and compositional video understanding tasks, this work reveals a stark dissociation between action recognition and the ability to count how many times an action is performed. The paper introduces a new benchmark composed of 446 long-form, human-annotated fitness clips to systematically probe this deficit and analyze whether improvements in counting translate to broader temporal understanding.

Benchmark Construction and Dataset Properties

PushupBench was meticulously curated from 34 publicly available full-length fitness videos by 11 diverse YouTube creators, covering 375 unique exercise types (84% exercise uniqueness across 446 clips).

Figure 1: Exercise type diversity in PushupBench; high diversity ensures generalization evaluation rather than mere memorization.

The average video duration is 36.7 seconds (range: 22–117s), explicitly challenging models to sustain temporal state tracking and detect action boundaries robustly. Annotation protocols ensured accurate, temporally-resolved repetition counts, with cross-validation for ambiguous instances to maintain label fidelity.

A key aspect is the distribution of ground truth counts, which is notably peaked at 10 repetitions—the dominant set size in human workouts.

Figure 2: Ground truth repetition count distributions in training and benchmark data, both mode at 10.

This real-world statistical property creates a confound exploited by weaker models, which collapse to constant prediction (typically “10”) rather than learning counting as temporal reasoning.

Evaluation Methodology and Metrics

The evaluation protocol standardizes VLM input by uniformly sampling frames at 5 fps (up to 112 frames per clip, 360p), with prompting diversity to avoid overfitting and maximize probing generalization. Three core metrics are reported:

• Exact Match: Proportion of predictions exactly matching ground truth.
• Mean Absolute Error (MAE): Average absolute difference between predictions and ground truth.
• $R^2$ (Coefficient of Determination): Quantifies whether the model tracks scaling with ground truth and penalizes both bias and variance; positive $R^2$ is indicative of genuine counting, while negative values reveal collapse to statistical priors.

Main Empirical Results

The central finding is that all evaluated models—frontier commercial VLMs (e.g., Gemini 3 Flash) and open-source VLMs (Qwen3-VL variants)—perform poorly at repetition counting. Gemini 3 Flash achieves the best result at only 42.1% exact accuracy and $R^2=0.82$. Open-source Qwen3-VL-4B is near random with an 8.2% exact match and negative $R^2$, comparable to a constant “10” baseline.

Figure 3: Predicted vs. ground truth for four models; Gemini 3 Flash tracks ground truth, while Qwen3-VL-4B collapses to constant prediction. Fine-tuning with DAPO improves $R^2$ and exact match, indicating real counting gains.

The underlying pathology is made explicit by $R^2$: Qwen3-VL-4B-Thinking (baseline) outputs “10” for virtually every input ($R^2=-0.34$), while Gemini 3 Flash demonstrates genuine (albeit noisy) tracking of ground truth ($R^2=0.82$). Models like TransRAC (supervised, video-only) and constant baselines confirm that exploiting dataset statistics enables modest “accuracy” without any temporal reasoning.

Fine-Tuning and Reward Design

Using just 391 carefully curated training samples (after excluding low-temporal-resolution and GT=10 mode-collapse samples), the Qwen3-VL-4B-Thinking model was fine-tuned via RL (VERL+DAPO, using a smooth scale-invariant linear reward). This minimal, high-quality training led to:

• Exact Match: 8.2% ⟶ 14.5%
• $R^2$: -0.34 ⟶ 0.38

Crucially, these improvements are not artifacts of statistical “hacks,” but true gains in sequence-wise temporal counting. The ablation demonstrates that data quality—matched to the Nyquist frame threshold and free of on-screen exploitable cues—is essential, outweighing naïve dataset scaling.

Figure 4: Reward function ablation reveals that Linear Percentage rewards enable learning despite early mode collapse; $R^2$ is maximized, distinguishing true counting from statistical exploitation.

Generalization to Broader Video Understanding

A critical claim supported by experiments is that counting is not a narrow or over-fitted task. Fine-tuning on repetition counting (391 samples) led to consistent improvements on unrelated, established video understanding benchmarks: MVBench (+2.45), PerceptionTest (+1.23), and TVBench (+2.75). Analysis indicates these gains are localized to temporal-reasoning subtasks, with spatial object counting often declining after counting training. This specificity suggests counting serves as a diagnostic and proxy for learning generic temporal reasoning, not just overfitting to fitness videos.

Error and Failure Analysis

The analysis reveals two main failure classes:

1. Mode Collapse: Models exploit the modal distribution, outputting “10” for all inputs, achieving non-trivial accuracy but failing $R^2$0.
2. Reward Hacking: During RL fine-tuning, VLMs may exploit on-screen counters or timers. Manual removal of visual clues and careful reward shaping are essential to mitigate shortcut behavior.

Figure 5: On-screen counter removal ensures models are evaluated on temporal reasoning rather than OCR shortcuts.

Figure 6: Examples of on-screen exercise parameters that can be exploited for reward hacking.

Practical and Theoretical Implications

From a systems evaluation perspective, PushupBench establishes that no current foundation VLM—open-source or proprietary—is reliable for fine-grained temporal counting in unconstrained video. The pronounced improvement on general video reasoning tasks following counting-focused RL post-training suggests new directions for curriculum design in VLM training, where counting (and similar fine-grained temporal reasoning probes) serve as transfer catalysts for broader competence.

Theoretically, these results highlight fundamental architectural gaps in current Transformer-based VLMs, particularly their failure to represent and aggregate fine-grained state transitions across extended sequences. The pronounced sensitivity to data curation and reward shaping further underscores the brittleness of current RLHF protocols for temporal tasks.

Future Directions

Extending pushup-style counting probes to other repetitive action domains (sports, industrial process videos, daily activities) will yield a broader diagnosis of VLM temporal competence. Further, scaling RL fine-tuning protocols and robust reward shaping techniques—potentially using dense, step-wise temporal supervision rather than aggregate sequence-level rewards—may be necessary for reliable counting and downstream generalization.

With the evidence that counting-specific RL post-training can notably transfer to unrelated temporal reasoning tasks, future VLM training curricula should incorporate diagnostic temporal counting modules as standard milestones for measuring and improving sequence-aware reasoning capabilities.

Conclusion

PushupBench rigorously demonstrates that current VLMs—including leading commercial and open-source models—are not proficient counters of repetitive actions in video. The benchmark exposes the inadequacy of exact match accuracy alone and the ease with which models can exploit dataset statistics, underscoring the necessity of metrics such as $R^2$1. Importantly, counting-focused fine-tuning yields transferable gains for temporal reasoning in general video understanding. PushupBench thus offers both a diagnostic suite and a curriculum lever for advancing the temporal capacities of future multimodal foundation models.