Gym-QA: Fine-Grained Sports VideoQA

• Gym-QA is a benchmark for evaluating multi-level question answering over gymnastics videos, emphasizing fine-grained action segmentation and domain-specific reasoning.
• It is organized along difficulty (Easy, Medium, Hard) and granularity (Event, Set, Element) axes to expose limitations of generic VideoQA models.
• The framework employs reactive and deliberative reasoning modes, integrating dynamic motion segmentation and knowledge graph matching to enhance complex sports video interpretation.

Searching arXiv for papers directly related to Gym-QA, sports VideoQA, and fitness/action-quality assessment. Gym-QA is a sports Video Question Answering benchmark introduced in “FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoning” (Chen et al., 15 Sep 2025). It is derived from the FineGym dataset and is designed to evaluate multi-level understanding of fine-grained, fast-motion gymnastics videos. In the surrounding literature, the label also sits near a broader research program concerned with gym and fitness understanding through video, multimodal sensing, action quality assessment, and natural-language feedback, although those adjacent systems are not identical to the benchmark itself (Zuo et al., 10 Aug 2025, Yin et al., 2 Jun 2025, Parmar et al., 2022).

1. Definition, scope, and nomenclature

Gym-QA denotes a benchmark for question answering over gymnastics video, not a reinforcement-learning environment. This distinction matters because the word “Gym” also appears in infrastructure papers such as “OpenAI Gym” (Brockman et al., 2016) and “qgym: A Gym for Training and Benchmarking RL-Based Quantum Compilation” (Linde et al., 2023), whose subject is environment standardization for RL rather than sports-video reasoning.

Within the FineQuest formulation, Gym-QA is positioned as a challenging sports VideoQA benchmark that probes whether a model can answer questions requiring multi-level understanding of fine-grained visual events in gymnastics. The benchmark is explicitly described as being derived from FineGym, and its role is to expose the limitations of generic and training-free VideoQA models when domain knowledge and multi-stage reasoning are absent (Chen et al., 15 Sep 2025).

A useful way to delimit the topic is to separate three layers. First, Gym-QA proper is the FineQuest benchmark. Second, there is a neighboring line of work on fitness and sports action quality assessment, exemplified by FLEX, Fitness-AQA, FineDiving-Pose, and PhysiQ (Yin et al., 2 Jun 2025, Parmar et al., 2022, Qi et al., 7 Jan 2025, Wang et al., 2022). Third, there are interactive feedback systems such as FormCoach, which approach gym understanding through comparative coaching rather than discrete QA alone (Zuo et al., 10 Aug 2025). This suggests that Gym-QA belongs to a wider technical ecosystem in which answering questions, detecting errors, and generating corrective feedback are increasingly treated as adjacent inference problems.

2. Benchmark structure and task axes

Gym-QA is organized along two explicit axes: question difficulty and action granularity. The reported difficulty levels are Easy, Medium, and Hard, while the action-granularity categories are Event, Set, and Element (Chen et al., 15 Sep 2025). These axes are central because they reveal where generic models fail and where domain-specific reasoning becomes necessary.

The benchmark is described as testing “multi-level” understanding of gymnastics video. In practice, that means a system is not only asked coarse questions that can be answered from global scene cues, but also finer questions that require attention to sub-actions and domain-specific terminology. The FineQuest summary emphasizes that the hardest questions are those at the Set and Element levels and those in the Medium and Hard difficulty bands, where prior models “collapse” and where knowledge-assisted reasoning provides the largest gains (Chen et al., 15 Sep 2025).

A compact view of the benchmark organization is as follows.

Axis	Categories	Role
Source dataset	FineGym	Fine-grained gymnastics video source
Difficulty	Easy, Medium, Hard	Varying query complexity
Granularity	Event, Set, Element	Multi-level action understanding

The structure implies that Gym-QA is not merely a recognition benchmark with paraphrased labels. A plausible implication is that it evaluates compositional sports understanding: temporal localization, fine-grained action disambiguation, and alignment between visual evidence and domain vocabulary.

3. FineQuest and the methodological treatment of Gym-QA

FineQuest addresses Gym-QA with a dual-mode framework inspired by cognitive science: Reactive Reasoning for straightforward queries and Deliberative Reasoning for more complex ones (Chen et al., 15 Sep 2025). In the reactive mode, a query-difficulty analysis determines whether the question can be handled directly by a zero-shot multimodal LLM. This path is intended for simple questions, such as coarse identification tasks.

The deliberative mode is activated for questions requiring multi-step reasoning or domain knowledge, which the FineQuest summary states accounts for much of the Medium/Hard and Set/Element space. Its pipeline has several explicit components. A Dynamic Motion Segmenter breaks the video into sub-action segments using motion intensity via optical flow and sliding-window analysis. A Key Clip Selector then filters the segments through hierarchical contrastive decoding, using spatial, temporal, and spatio-temporal perturbations to isolate clips aligned with the query. A Captioner generates rich descriptions of the selected clips, and a Fine-grained Matcher aligns captions and visual clip embeddings to SSGraph, a multimodal sports knowledge scene graph spanning nine sports (Chen et al., 15 Sep 2025).

SSGraph plays the knowledge-augmentation role. It stores both visual instances and domain-specific terminology, and the top matched nodes are inserted into the prompt for the reasoner. The FineQuest summary states that this is especially important for Gym-QA questions that require mapping visual evidence to precise gymnastics concepts, including element-level terminology (Chen et al., 15 Sep 2025).

An ablation reported for Gym-QA underscores the dependence on this knowledge-matching stage. FineQuest in full form yields Easy/Medium/Hard/Overall scores of 80.1/50.0/47.9/57.0, whereas removing the fine-grained matcher reduces those scores to 79.8/37.9/12.9/41.9 (Chen et al., 15 Sep 2025). The sharp drop on Hard questions indicates that the benchmark is highly sensitive to domain grounding rather than only to generic visual-language fluency.

4. Empirical performance and what it reveals

FineQuest reports large improvements on Gym-QA relative to prior training-free systems. Against LLaVA-Next-Video, the reported scores move from 87.9 to 88.2 on Event, from 47.2 to 64.6 on Set, and from 12.1 to 58.4 on Element; the corresponding Action Overall score rises from 49.1 to 70.4. On the full difficulty split, the reported numbers move from 80.8/38.4/13.1/42.5 to 81.0/58.9/49.9/62.1 for Easy/Medium/Hard/Overall (Chen et al., 15 Sep 2025).

Model	Event	Set	Element	Action Overall	Full Overall
LLaVA-Next-Video	87.9	47.2	12.1	49.1	42.5
FineQuest	88.2	64.6	58.4	70.4	62.1

The pattern is more informative than the raw averages. FineQuest gains only marginally on Event and Easy questions, but the improvements are large on Set, Element, Medium, and Hard. That asymmetry indicates that Gym-QA is most diagnostic when the query cannot be solved by coarse video semantics alone. In other words, the benchmark primarily stresses fine-grained temporal reasoning and domain knowledge transfer, not merely generic video captioning capacity.

FineQuest is also reported to outperform VideoTree by large margins, especially in the Set/Element and Medium/Hard regimes (Chen et al., 15 Sep 2025). This makes Gym-QA a discriminative benchmark for training-free reasoning pipelines: it separates approaches that can only summarize salient frames from those that can segment, retrieve, and reason over sports-specific structure.

5. Relation to adjacent gym and fitness QA research

Although Gym-QA is a sports VideoQA benchmark, nearby research broadens the notion of “gym QA” toward coaching, assessment, and multimodal interpretation. FormCoach is the clearest example on the language-feedback side. It benchmarks VLMs on 1,700 expert-annotated user-reference video pairs spanning 22 strength and mobility exercises, with rubric-based evaluation over Accuracy, Actionability, and Hallucination (Zuo et al., 10 Aug 2025). The best reported model, GPT-4.1, attains 58.2 accuracy, 94.4 actionability, and 74.3 hallucination. This creates an important contrast with Gym-QA: a system can produce fluent, actionable-sounding feedback while still misidentifying form errors at a high rate.

On the multimodal action-quality side, FLEX extends fitness assessment beyond RGB-only settings. It contains 20 weight-loaded fitness actions performed by 38 subjects across 3 skill levels for 10 repetitions each, with 5-view RGB video, 3D pose, sEMG, and physiological information (Yin et al., 2 Jun 2025). FLEX also introduces a knowledge-graph-based annotation scheme over actions, action keysteps, error types, and feedback, together with a penalty-based score:

$$\text{Score} = \left[1 - \frac{\sum_{i=1}^N W_i}{\sum_{j=1}^M W_j}\right] \times 100$$

Its baseline results show that moving from single view to multiview improves Spearman’s $\rho$ from 0.81 to 0.90 and reduces $R$-$l2$ from 1.76 to 0.91, with further gains from pose and sEMG features (Yin et al., 2 Jun 2025). This suggests that future Gym-QA-like systems may benefit from multimodal supervision even if the current benchmark is video-centric.

Earlier work also develops foundational representations for workout-form understanding. Fitness-AQA curates a dataset of BackSquat, BarbellRow, and OverheadPress videos annotated by expert trainers for fine-grained exercise errors, and proposes domain-knowledge-informed self-supervised learning through pose contrastive learning and motion disentangling (Parmar et al., 2022). PhysiQ, by contrast, uses smartwatch IMU signals to quantify off-site physical-therapy exercise quality in terms of range of motions, stability, and repetition, combining absolute classification with relative similarity comparison in a Siamese architecture (Wang et al., 2022). These systems are not Gym-QA benchmarks, but they define adjacent target variables that Gym-QA-style question answering may eventually need to verbalize or justify.

A related strand focuses on action quality assessment in sports rather than open-ended QA. The hierarchically pose-guided multi-stage contrastive regression method for AQA introduces a FineDiving-Pose dataset with 12,722 manually annotated frames and 288,000 automatically annotated frames, and reports strong results on FineDiving and MTL-AQA (Qi et al., 7 Jan 2025). This work reinforces the broader point that fine-grained sports understanding often depends on explicit sub-action segmentation and pose structure, which is consonant with the design of FineQuest’s deliberative pipeline.

6. Limitations, misconceptions, and research directions

A common misconception is that high-quality language output implies reliable gym understanding. The FormCoach results argue against that interpretation: the best reported model is highly actionable in wording yet still reaches only 58.2 accuracy and exhibits a 74.3 hallucination rate (Zuo et al., 10 Aug 2025). For Gym-QA, this is a caution against equating answer fluency with evidential correctness.

Another limitation concerns sensing modality. Visual-only systems face ambiguity from occlusion, suboptimal viewpoints, lack of depth, and compound-movement complexity, all of which are explicitly identified in FormCoach (Zuo et al., 10 Aug 2025). FLEX responds by adding 3D pose, sEMG, heart rate, and respiration (Yin et al., 2 Jun 2025), while PhysiQ relies on wearable inertial sensing (Wang et al., 2022). A plausible implication is that benchmark progress on Gym-QA will eventually intersect with multimodal fusion, especially for fine-grained or safety-critical queries.

The FineQuest ablations also indicate that benchmark gains are concentrated in the most knowledge-intensive strata (Chen et al., 15 Sep 2025). This means Gym-QA is not simply measuring stronger generic vision-LLMs; it is measuring whether a system can operationalize structured sports knowledge during inference. In that respect, SSGraph in FineQuest and the knowledge graph in FLEX point in a similar direction: both make domain structure explicit rather than leaving it implicit in model parameters (Chen et al., 15 Sep 2025, Yin et al., 2 Jun 2025).

More broadly, the present landscape suggests a convergence of three research agendas. One is benchmarked sports VideoQA, exemplified by Gym-QA. A second is fitness and sports AQA, where scoring and phase-wise error localization are central. The third is interactive coaching, where answers must be concise, actionable, and safe. Gym-QA presently belongs most directly to the first agenda, but adjacent work indicates that future systems may be evaluated not only on whether they answer correctly, but also on whether they localize the relevant sub-action, ground the answer in domain knowledge, avoid hallucination, and support corrective feedback (Chen et al., 15 Sep 2025, Zuo et al., 10 Aug 2025, Yin et al., 2 Jun 2025).