Multi-Agent System for Comprehensive Soccer Understanding (2505.03735v1)

Abstract: Recent advancements in AI-driven soccer understanding have demonstrated rapid progress, yet existing research predominantly focuses on isolated or narrow tasks. To bridge this gap, we propose a comprehensive framework for holistic soccer understanding. Specifically, we make the following contributions in this paper: (i) we construct SoccerWiki, the first large-scale multimodal soccer knowledge base, integrating rich domain knowledge about players, teams, referees, and venues to enable knowledge-driven reasoning; (ii) we present SoccerBench, the largest and most comprehensive soccer-specific benchmark, featuring around 10K standardized multimodal (text, image, video) multi-choice QA pairs across 13 distinct understanding tasks, curated through automated pipelines and manual verification; (iii) we introduce SoccerAgent, a novel multi-agent system that decomposes complex soccer questions via collaborative reasoning, leveraging domain expertise from SoccerWiki and achieving robust performance; (iv) extensive evaluations and ablations that benchmark state-of-the-art MLLMs on SoccerBench, highlighting the superiority of our proposed agentic system. All data and code are publicly available at: https://jyrao.github.io/SoccerAgent/.

• The paper introduces a holistic framework featuring SoccerWiki, SoccerBench, and SoccerAgent to advance soccer analytics.
• The paper details a modular multi-agent system with planning and execution components that leverage 18 specialized tools for multi-modal reasoning.
• The paper demonstrates superior performance on TextQA and VideoQA tasks compared to state-of-the-art models, highlighting its practical impact for sports analytics.

This paper introduces a comprehensive framework for holistic soccer understanding, addressing the limitations of existing AI research which often focuses on isolated tasks and lacks deep domain knowledge. The authors present three main contributions: SoccerWiki, a large-scale multimodal soccer knowledge base; SoccerBench, a comprehensive soccer-specific benchmark; and SoccerAgent, a novel multi-agent system designed for collaborative reasoning.

SoccerWiki: This is the first large-scale multimodal knowledge base specifically for soccer. It integrates extensive information from sources like Wikipedia and Flashscore, covering 9,471 players, 266 teams, 202 referees, and 235 venues. Each entity includes images and detailed attributes such as career statistics, personal profiles, and team histories. It also incorporates detailed game information from 1,988 matches from major European leagues and championships. This knowledge base is crucial for enabling knowledge-driven reasoning beyond simple visual perception tasks.

SoccerBench: To provide a standardized and comprehensive evaluation platform, the authors constructed SoccerBench. It is described as the largest and most comprehensive soccer-specific benchmark to date, featuring around 10,000 multi-choice question-answering pairs. These QA pairs are curated across 13 distinct soccer understanding tasks (summarized in Table 1), covering background knowledge, match situations, camera status, jersey information, replay grounding, action classification, commentary generation, and foul recognition. The benchmark integrates SoccerWiki with various existing soccer datasets. The data curation pipeline involves generating open-ended QA pairs using predefined templates or prompting LLMs, followed by converting them into multi-choice format with plausible distractors. The benchmark includes TextQA, ImageQA, and VideoQA tasks, offering diverse complexity levels and modalities for robust evaluation. Practical implementation involves using the provided multimodal QA pairs, where models are given a question and multi-choice options and must select the correct answer.

SoccerAgent: To tackle the challenging and knowledge-intensive tasks in SoccerBench, the paper proposes SoccerAgent, a novel multi-agent system. SoccerAgent features a modular architecture designed for comprehensive analysis and precise responses to multimodal soccer questions. It consists of two primary components: a planning agent ($\mathcal{A}_{plan}$) responsible for decomposing complex questions into sequential sub-tasks and constructing an optimal tool chain, and an execution agent ($\mathcal{A}_{exec}$) that iteratively processes the planned tool chain. The system leverages a toolbox of 18 specialized tools, comprising 12 soccer-specific tools (derived from pre-trained models like UniSoccer and Qwen2.5-VL, or built upon SoccerWiki for retrieval) and 6 general-purpose multimodal parsing tools (leveraging models like CLIP and GroundingDINO, or general LLMs like DeepSeek-v3). The execution agent is history-aware, adapting tool inputs based on previous steps. Tool calls adhere to a strict format with <Call> and <EndCall> markers and specific delimiters for tool name, query, material path, and purpose, ensuring interpretability and robustness. The tool chain planning and execution process is detailed, showing how the system orchestrates different tools to answer complex questions. For practical implementation, this architecture implies setting up a central agent core capable of managing tool interactions and processing multimodal inputs/outputs. The toolbox requires integrating various specialized models and APIs, many of which are open-source as detailed in the paper.

Experiments and Results: The paper evaluates SoccerAgent against various state-of-the-art MLLMs, including commercial APIs (Claude 3.7 Sonnet, GPT-4o, Gemini 2.0 Flash) and open-source models (DeepSeek-v3/R1, Qwen2.5-VL, LLaVA variants, VideoLLaMA3, VideoChat). Results on SoccerBench (Table 2) demonstrate that the benchmark effectively differentiates model capabilities. SoccerAgent, despite generating open-ended answers before selecting from multiple choices (without seeing options during reasoning), achieves superior performance on tasks requiring specific soccer knowledge (Q1/4, Q9) and leads overall in TextQA and VideoQA, while remaining competitive in ImageQA. This highlights the practical advantage of the agentic system with specialized tools and integrated knowledge. Ablation studies (Table 3) show that providing task descriptions improves accuracy across most question types, while providing execution examples has mixed effects, suggesting the execution agent's inherent capability with visual information. Qualitative results (Figure 4) illustrate the step-by-step reasoning and tool execution process, demonstrating SoccerAgent's error correction capabilities when a planned tool call fails.

Implementation Considerations: Implementing SoccerAgent involves integrating various open-source and potentially commercial models as tools. Computational requirements depend heavily on the complexity and size of the models used in the toolbox. Deploying such a system requires managing the orchestration logic of the agents and ensuring smooth data flow between different tools and modalities. The SoccerWiki knowledge base needs to be stored and indexed efficiently for retrieval tools. The paper provides detailed prompts and tool descriptions in the appendix, which are valuable for anyone attempting to reproduce or extend this work. The public availability of data and code is a significant advantage for practical application and further research in this area. The modular nature of the agent system allows for potential scaling by replacing or adding specialized tools as needed.

In summary, the paper provides a practical framework, a challenging benchmark, and a robust multi-agent system for comprehensive soccer understanding, laying a strong foundation for future knowledge-driven sports analytics.