Towards Automatic Soccer Commentary Generation with Knowledge-Enhanced Visual Reasoning

Abstract: Soccer commentary plays a crucial role in enhancing the soccer game viewing experience for audiences. Previous studies in automatic soccer commentary generation typically adopt an end-to-end method to generate anonymous live text commentary. Such generated commentary is insufficient in the context of real-world live televised commentary, as it contains anonymous entities, context-dependent errors and lacks statistical insights of the game events. To bridge the gap, we propose GameSight, a two-stage model to address soccer commentary generation as a knowledge-enhanced visual reasoning task, enabling live-televised-like knowledgeable commentary with accurate reference to entities (players and teams). GameSight starts by performing visual reasoning to align anonymous entities with fine-grained visual and contextual analysis. Subsequently, the entity-aligned commentary is refined with knowledge by incorporating external historical statistics and iteratively updated internal game state information. Consequently, GameSight improves the player alignment accuracy by 18.5% on SN-Caption-test-align dataset compared to Gemini 2.5-pro. Combined with further knowledge enhancement, GameSight outperforms in segment-level accuracy and commentary quality, as well as game-level contextual relevance and structural composition. We believe that our work paves the way for a more informative and engaging human-centric experience with the AI sports application. Demo Page: https://gamesight2025.github.io/gamesight2025

• The paper presents a two-stage framework (GameSight) that decouples visual reasoning for precise player alignment from knowledge-enhanced commentary refinement.
• It leverages detailed visual and contextual cues, achieving up to 96.3% accuracy in human-level entity resolution and an 18.5% improvement over baseline models.
• The system integrates external statistics and dynamic match data to produce broadcast-quality commentary that closely mimics human narrative styles.

Knowledge-Enhanced Visual Reasoning for Soccer Commentary Generation with GameSight

Introduction and Motivation

Automatic soccer commentary generation has traditionally been approached through end-to-end models that output anonymized, live-text commentary lacking explicit references to in-game entities, context, and statistical depth. Such methods are insufficient for replicating the type of informative, entity-rich commentary found in live televised broadcasts, which integrates precise player and team references, evolves with game context, and incorporates historical statistics.

The paper "Towards Automatic Soccer Commentary Generation with Knowledge-Enhanced Visual Reasoning" (2604.00057) introduces GameSight, a comprehensive two-stage framework for commentary generation formulated as a knowledge-enhanced visual reasoning problem. GameSight advances the field by jointly solving entity-grounded alignment and knowledge integration, thereby producing contextually and statistically rich commentary aligned with human broadcast standards.

Figure 1: Overview of the GameSight architecture, illustrating joint use of visual and contextual sources for entity-aligned, knowledge-enhanced soccer commentary generation.

Problem Formulation and System Architecture

The central challenge is formulated as follows: given a short soccer video segment $\mathcal{S}$ and an anonymized, coarse commentary $C_A$, the objective is to output final commentary $C_{KE}$ that is precisely entity-grounded and contextually informed.

GameSight introduces a decoupled, two-stage system:

1. Stage I: Visual Reasoning for Entity Alignment This stage focuses on resolving entity anonymization by leveraging detailed visual cues (faces, jersey numbers), contextual match data (line-ups, events), and temporal knowledge guidance for fine-grained grounding.
2. Stage II: Commentary Refinement with Knowledge Once entities are disambiguated, external historical statistics and dynamic internal match context are injected to further enrich the commentary, yielding outputs analogous in style and information density to human commentators.

   Figure 2: Two-stage system: Stage I aligns entities using visual and contextual cues, Stage II injects statistical and match-specific knowledge into commentary.

Visual Reasoning for Entity Alignment

Empirical Basis

A core empirical finding, established via a "Player Guessing" human baseline study, is that accurate entity alignment demands more than event-driven player tracking or pure frame analysis. Human annotators achieved 96.3% accuracy, with over 84.5% of correct guesses involving complex cues (e.g., close views, player faces, context events).

Figure 3: Human reliance on multifaceted cues demonstrates the necessity of combining visual and contextual information for robust player alignment.

Fine-grained Visual and Contextual Cues

Broadcast soccer videos encompass various shot types—long, medium, close-up, and out-of-field views—each offering different levels of player identity signal.

Figure 4: Distribution of soccer game view shots by time and example keyframes, illustrating the diversity of available identity cues.

The system applies fine-grained shot analysis:

• Detects shot boundaries and classifies view type for triage.
• Recognizes faces and numerals in close-up frames, yielding high-confidence identity predictions.
• Associates visual cues with dynamic line-ups and current match state.

Visual reasoning is operationalized in a chain-of-thought (CoT) paradigm, allowing compositional inference over multifaceted evidence (e.g., jersey recognition in close views, timeline context for rapidly evolving events).

Figure 5: CoT procedure sample—stepwise reasoning integrates scene, event, player, and history clues for precise entity assignment.

Temporal Event Grounding

Temporal segmentation is non-trivial; events often span multiple frames with variable delays in the associated close-up shots. A Q-former-based cross-attention mechanism identifies frame-to-event alignment by weighing each frame's contribution to commentary, leading to robust grounding even in complex multi-event segments.

Figure 6: Test set frame-level distribution—left: event grounding peaks at timestamp centers; right: valid close views cluster near events, providing strong identity signals.

Training Regimen

Supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) are jointly employed to enhance the backbone Video-LMM (Qwen2.5-VL). Ablation indicates substantial contributions from close view analysis, face and jersey cues, and CoT chaining, with accuracy improvements exceeding 27%, and a final player alignment improvement of 18.5% over leading proprietary models.

Knowledge-Enhanced Commentary Generation

Knowledge Source Taxonomy

The system incorporates two knowledge streams for refinement:

1. External Statistics: Drawn from an enhanced SoccerRAG-based retrieval system (SoccerKAG), supporting fine-grained, temporally-consistent queries for both player and team-level statistics.
2. Internal Game Context: Iteratively updated database capturing match-internal events (line-up changes, score, disciplinary record) and recent event history, paralleling human commentator referencing practices.

   Figure 7: Event-level breakdown—knowledge enrichment (e.g., statistical references) is unevenly distributed across events, peaking for goals, cards, and corners.

Commentary Structure and Synthesis

The system induces a structure in commentary production:

• Description: Raw event narration.
• Explanation: Contextual or tactical analysis, often supported by historical statistics or situational context.
• Comment: Expressive or evaluative lines, mirroring human emphasis and emotional engagement.

Human-broadcast parity is achieved by maintaining the descriptive fraction below 50% and blending explanatory and commentary elements.

Figure 8: Structural composition—GameSight commentary mimics televised baselines with increased explanation and comment, surpassing live-text models in analytical depth.

Experimental Results

Entity Alignment

On the SN-Caption-test-align benchmark, GameSight—using Qwen2.5-VL with SFT+GRPO—achieves 71.1% player alignment accuracy, outperforming Gemini 2.5-pro by 18.5%. Detailed ablation supports the need for multi-modal clues and Chain-of-Thought reasoning over vanilla inference.

Commentary Quality and Knowledge Injection

GameSight demonstrates competitive or superior BLEU, METEOR, ROUGE, and CIDEr across both textstyle and televised ground truths, with pronounced improvements when judged against human-televised references. For knowledge accuracy, external accuracy improves to 81.8% (SoccerKAG, +16.11%), and internal statistics accuracy reaches 98.76% on goal events and 90.74% on other contexts.

Game-level analysis using Coh-Metrix validates high discourse coherence and contextual relevance. Human studies (MOS scoring) confirm significant subjective preference for GameSight output.

Practical and Theoretical Implications

GameSight operationalizes a robust two-stage paradigm for automatic, knowledge-rich sports commentary. The explicit decoupling of visual entity alignment from knowledge-driven narrative refinement sets a robust pathway for future extension to other structured domains, where entity resolution and context-aware explanation are both critical.

Practically, this work signals increased viability for AI-driven sports media, fan engagement platforms, and live content augmentation—where high-fidelity, low-latency, and entity-grounded narrative is essential. The framework also highlights limits in streaming use and the importance of robust timestamp alignment and diverse knowledge integration.

Theoretically, this architecture exemplifies advances in multi-modal neural reasoning, compositional alignment, and retrieval-augmented generation, augmenting LMM capabilities in complex, real-time information environments.

Limitations and Prospects

While GameSight demonstrates marked improvement over single-stage or non-knowledge methods, open challenges remain:

• Transferability to other sports or event domains given soccer-specific priors.
• Full realization of real-time, streaming processing without loss of entity or knowledge fidelity.
• Incorporation of more diverse knowledge sources (e.g., health status, social sentiment).
• Achieving true human-level intuition, creativity, and nuanced expression in automated commentary.

Conclusion

GameSight substantially advances automatic soccer commentary generation by integrating entity-grounded visual reasoning with dynamic knowledge augmentation. The approach yields commentaries that better match human expectations for informativeness, context, and engagement. The staged architecture and empirical evaluations provide a methodological and technical baseline for further multimodal, knowledge-intensive generation tasks in sports and beyond.

(2604.00057)