FineQuest: Training-Free Sports VideoQA

• FineQuest is a training-free framework that combines adaptive dual-mode reasoning with a specialized sports knowledge scene graph to enhance fine-grained VideoQA.
• The framework employs both Reactive and Deliberative Reasoning, using motion segmentation and contrastive clip selection to address structural challenges in sports videos.
• Empirical results on Gym-QA and Diving-QA benchmarks show significant improvements on complex, element-level queries while retaining strong general VideoQA capabilities.

FineQuest is a training-free framework for sports video question answering that addresses the mismatch between general-purpose multimodal LLMs and the fine-grained, temporally dense, terminology-heavy character of sports video understanding. It combines a dual-mode reasoning policy—Reactive Reasoning for straightforward queries and Deliberative Reasoning for complex ones—with SSGraph, a multimodal sports knowledge scene graph spanning nine sports. The framework is introduced together with two sports VideoQA benchmarks, Gym-QA and Diving-QA, and is reported to achieve state-of-the-art performance on these benchmarks and on SPORTU while maintaining strong general VideoQA capability (Chen et al., 15 Sep 2025).

1. Domain and problem formulation

FineQuest is situated in sports VideoQA, a setting in which standard long-video or general VideoQA pipelines are reported to fail for reasons that are structural rather than merely data-scarcity related. The paper identifies five main difficulties: fast, fine-grained motion; multi-camera and frequent shot switches; multi-object and context-rich interactions; domain-specific terminology; and the lack of large annotated sports QA data (Chen et al., 15 Sep 2025).

These constraints make sports videos qualitatively different from ordinary web or instructional videos. A correct answer may depend on subtle body posture transitions, the apparatus or court context, the event/set/element hierarchy of a sport, or terminology that a generic vision-LLM can paraphrase only coarsely. The paper argues that a model may describe “an athlete doing a jump” while still failing to map that observation to a specialized sports concept.

The work is also framed as a critique of existing training-free VideoQA systems such as VideoTree and related VLM+LLM pipelines. In the reported analysis, scene clustering based on methods such as $k$-means is a poor fit for sports footage because camera switches do not necessarily correspond to semantic action boundaries; keyframe selection misses motion semantics that reside between frames; general VLMs lack sports knowledge; and a uniform reasoning strategy is inadequate because simple queries and fine-grained multi-step queries require different computational pathways. This motivates FineQuest’s adaptive, query-conditioned design.

2. Dual-mode reasoning architecture

FineQuest is organized as an “agent-of-thoughts” system with a system switch between a fast path and a slow path. The architecture comprises a Reactive Reason Agent, a Dynamic Motion Segmenter, a Key Clip Selector, a Fine-grained Matcher, and a Reasoner (Chen et al., 15 Sep 2025).

Component	Function	Role in pipeline
Reactive Reason Agent	Query difficulty analysis	Chooses direct answer or system switch
Dynamic Motion Segmenter	Adaptive sub-action proposal generation	Replaces scene clustering
Key Clip Selector	Question-relevant clip filtering	Narrows evidence set
Fine-grained Matcher	SSGraph-based knowledge injection	Aligns clips with sports semantics
Reasoner	Final answer generation	Produces response from selected evidence

The paper formalizes generic VideoQA as

$$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$

where $\mathcal{V}$ is the video, $\mathcal{Q}$ the query, $\mathcal{E}_V$ and $\mathcal{E}_L$ the video and language encoders, and $\mathcal{F}$ the multimodal visual agent. Reactive Reasoning is written as

$$\mathcal{R}_{\text{React}}=\mathcal{F}_{\text{React}}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right].$$

Reactive Reasoning is explicitly linked to “System 1” thinking. It evaluates a query along four axes: question-video relevance, question type as static versus dynamic, reasoning requirement as single-step versus multi-step, and external knowledge dependency. If the query is judged simple, the system returns an answer directly with a brief reasoning basis. If the query is judged complex, the system emits a system switch and activates Deliberative Reasoning.

Deliberative Reasoning corresponds to “System 2” thinking. It is invoked for queries that require temporal decomposition, sub-action reasoning, key-clip selection, or sports-domain knowledge. Its three reported stages are motion segmentation, key-clip selection, and SSGraph-assisted fine-grained matching before final reasoning. The architecture therefore treats query complexity not as a prompt-engineering nuisance but as a control variable governing the entire inference trajectory.

3. Motion-aware segmentation and contrastive clip selection

The first stage of the deliberative path is the Dynamic Motion Segmenter, which is designed to segment sports footage into sub-action proposals without predefined timestamps. The reported intuition is that athletic motion intensity fluctuates naturally and that brief pauses between action sets can be detected from motion magnitude rather than from scene appearance (Chen et al., 15 Sep 2025).

Before segmentation, FineQuest uses SAM 2 to mask or highlight athletes while preserving background context. It then computes optical flow between adjacent frames, converts the flow into a motion score sequence, and applies a sliding-window procedure. Within each window, it computes the mean and standard deviation of motion and sets a dynamic threshold

$$\text{threshold} = \text{mean} - z \times \text{std},$$

where $z$ is adaptively adjusted based on local variability. If the current motion falls below the threshold and clip-length constraints are satisfied, a proposal boundary is created. The pseudocode also enforces a minimum clip length using clip_len and adapts $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$0 using a z_range. The result is a proposal list intended to reflect sub-actions rather than camera edits.

The Key Clip Selector addresses a second failure mode of prior training-free systems: direct frame-query similarity can hallucinate or over-rely on language priors. FineQuest instead uses contrastive decoding on clips. Given a clip $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$1, a query $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$2, and an augmented query $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$3, the contrastive distribution is defined as

$$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$4

The paper extends this to three distortions: spatial distortion by Gaussian noise, temporal distortion by temporal warping, and spatio-temporal distortion by combining both. The reported default weights are $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$5, $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$6, and $$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$7. The top-$$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$8 clips by relevance are selected, and adjacent clips may be merged.

The appendix reports that Gaussian noise works best spatially, temporal warping works best temporally, all-shuffle is worst because it destroys sequence structure, and CutMix harms fine-grained recognition because it breaks spatial integrity. This suggests that FineQuest’s clip-selection stage is not merely a retrieval heuristic but a calibrated mechanism for preserving motion semantics under adversarial perturbation.

4. SSGraph and fine-grained sports knowledge matching

SSGraph is the principal knowledge component of FineQuest. It is described as a multimodal sports knowledge scene graph and as the first sports-specific KSG / MMKG hybrid, spanning nine sports: gymnastics, diving, basketball, soccer, ice hockey, tennis, baseball, badminton, and volleyball (Chen et al., 15 Sep 2025).

The graph stores tuples of the form

$$\mathcal{R}=\mathcal{F}\left[\mathcal{E}_V(\mathcal{V}),\mathcal{E}_L(\mathcal{Q})\right],$$9

linking nodes through relations. Its contents are divided into two broad categories. The first is visual instances, including objects, sub-actions, movement instances, attributes, and visual proposals. The second is domain-specific terminology and rules, including rule terms, scoring criteria, event/set/element hierarchy, relational dependencies, contextual dependencies, and specialized action labels. SSGraph therefore functions as a sports semantics graph rather than a generic object graph.

The reported construction pipeline has four stages. Raw data collection uses FineGym, FineDiving, MultiSports, and internet sources. Knowledge processing collects videos for the finest-grained sport elements, selects high-definition videos with low camera-switch frequency and good lighting, manually watches videos multiple times, writes detailed motion descriptions, and supplements them with external references to ensure professional terminology. Node connection organizes the sports hierarchically—for example, Gymnastics $\mathcal{V}$0 event $\mathcal{V}$1 set $\mathcal{V}$2 element—and links elements with video and description nodes. Scene representation then extends the knowledge graph into a spatiotemporal scene graph with frame-wise graphs, subject-predicate-object triplets, and temporal coreference edges linking objects across frames.

The Fine-grained Matcher uses SSGraph at two levels. At the cross-modal instance level, captions and selected clips are aligned with SSGraph nodes corresponding to visual instances and textual descriptions. The paper states that Long-CLIP is used to compare text/text and video/video similarity, and that the top five sub-actions by similarity are selected for further cross-matching. At the relational scene level, the matcher reasons over spatial, action, causal, and temporal relations. For positive and negative relation phrasing, the final relational score is

$\mathcal{V}$3

The use of negative relation modeling is reported to reduce false matches and improve disambiguation, especially for relation-sensitive queries.

5. Benchmarks and evaluation protocol

FineQuest is introduced together with two new sports VideoQA benchmarks, Gym-QA and Diving-QA, derived from FineGym and FineDiving respectively (Chen et al., 15 Sep 2025).

Gym-QA is built from FineGym segmented into 6,031 event-level clips and includes clips containing 1 to 32 element actions. Its total size is 27,469 QA pairs. Diving-QA is built from FineDiving; the videos correspond to 31 action types, each including 3 to 4 sub-actions, for a total of 1,055 QA pairs.

The annotation procedure is explicitly human-centric rather than purely LLM-generated. Annotators are trained on guidelines, questions and answers are created in separate stages, invalid questions are refined or discarded, and every question must be answerable from the video itself. Open-ended questions are converted to multiple-choice format using GPT-4o to generate three distractors that are semantically coherent, unique, and distinct from the correct answer.

The benchmarks are structured along two difficulty taxonomies. The “full set” contains Easy questions for coarse understanding, Medium questions for finer comprehension, and Hard questions requiring domain-specific sports knowledge. The “action set” separates Event, Set, and Element levels, with Element being the finest-grained movement detail. This multi-axis design makes the benchmarks diagnostic rather than purely aggregate.

The evaluation protocol is zero-shot and training-free. FineQuest is applied to two 7B MLLMs, Video-LLaVA and LLaVA-Next-Video, to keep the backbone constant across comparisons. The main baseline is VideoTree, with additional reference models including VideoChatGPT, VideoChat2, and Tarsier. To assess whether sports specialization degrades generic capability, the paper also evaluates on MSVD-QA, MSR-VTT-QA, TGIF-QA, and ActivityNet-QA. Reported implementation details include one NVIDIA H100 GPU and adaptive clip sampling parameters $\mathcal{V}$4 and $\mathcal{V}$5 that depend on video length.

6. Empirical results and ablation findings

The reported empirical results show substantial gains on sports benchmarks, especially as question difficulty increases (Chen et al., 15 Sep 2025).

Using LLaVA-Next-Video as the base model, FineQuest improves Gym-QA overall from 46.8 for VideoTree to 70.4 and Diving-QA overall from 41.2 to 62.1. Using Video-LLaVA as the base model, Gym-QA improves from 41.5 to 63.6 and Diving-QA from 40.5 to 57.0. On SPORTU, LLaVA-Next-Video improves from 63.7 to 76.1 and Video-LLaVA from 60.0 to 73.2.

The largest gains occur on harder, finer-grained queries. With LLaVA-Next-Video on Gym-QA, Event-level accuracy changes from 87.9 to 88.2, Set-level from 44.0 to 64.6, and Element-level from 8.6 to 58.4. This pattern is central to the paper’s claim: the framework’s advantage is not primarily in coarse sport recognition but in high-resolution action and terminology grounding.

FineQuest is also reported to preserve or improve generic VideoQA performance. With LLaVA-Next-Video, MSVD-QA rises from 73.6 to 77.2, MSR-VTT-QA from 57.9 to 58.4, TGIF-QA from 71.4 to 72.0, and ActivityNet-QA from 53.5 to 55.3. With Video-LLaVA, MSVD-QA rises from 71.8 to 73.8, MSR-VTT-QA remains 59.0, TGIF-QA rises from 48.4 to 49.1, and ActivityNet-QA rises from 45.3 to 47.8.

The ablations isolate the contribution of each module. Removing the reactive mode lowers performance, especially on Easy questions, indicating that not all inputs benefit from heavy deliberation. Removing the Dynamic Motion Segmenter degrades Medium and Hard performance, consistent with the claim that motion-aware segmentation is essential in sports footage. Removing the Key Clip Selector reduces performance further, particularly on Hard questions. Removing the Fine-grained Matcher causes the largest collapse: the reported overall score drops from 57.0 for full FineQuest to 41.9 when the matcher is removed.

Additional analyses reinforce the same decomposition. On FineGym, temporal localization over 100 samples shows better mAP at both event and set levels than clustering-based segmentation. Negative relation modeling improves all relation types, with spatial relations benefiting the most. Adversarial evaluation created by mixing gymnastics and diving segments shows smaller degradation than Video-LLaVA or VideoTree. The framework is also reported to generalize to Olympic categories such as figure skating and table tennis.

7. Interpretation, significance, and limitations

FineQuest is presented as a training-free system, but its design shows that “training-free” does not mean knowledge-free or structure-free. The framework relies on SSGraph, explicit query-difficulty routing, motion-aware proposal generation, contrastive selection, and staged reasoning. This suggests that, in the paper’s usage, training-free refers to the absence of sports QA fine-tuning rather than the absence of domain-specific engineering (Chen et al., 15 Sep 2025).

Its broader significance lies in how it redefines sports VideoQA as a problem of adaptive inference under domain knowledge constraints. The framework treats simple queries as opportunities for low-latency direct response and complex queries as requiring temporal decomposition and semantic grounding. That distinction is closely tied to the reported performance pattern, where improvements are small for coarse Event-level recognition but very large for Set- and Element-level questions.

The paper also positions SSGraph as a domain memory that converts vague clip captions into precise sports concepts. A plausible implication is that FineQuest’s main contribution is not only better answer generation but also a more explicit interface between perception and formalized sports semantics. In that sense, it addresses a recurrent failure mode of general multimodal models: the inability to bind visual evidence to specialized ontologies.

The reported limitations are practical rather than conceptual. SSGraph construction still depends on specialized human effort, even though the pipeline uses rule-based filtering and LLM assistance. The framework is training-free but introduces multiple modules and prompts, which may increase engineering complexity and inference cost relative to a single-pass model. The appendix further identifies future extensions beyond question answering into sports commentary, judging or refereeing assistance, and broader sports analytics, along with future work on expanding the domain-specific knowledge base, incorporating more sophisticated reasoning mechanisms, and improving interpretation of more complex sports scenarios.