MARS2 2025: Multimodal Reasoning Benchmark

• MARS2 2025 Challenge is a benchmark for real-world multimodal reasoning that integrates both spatial and abstract reasoning in diverse scenarios.
• It leverages two tailored datasets, Lens and AdsQA, to assess scene perception in daily settings and creative advertisement interpretation.
• The challenge uses three specialized tracks with distinct metrics to expose model limitations and drive methodological advances in multimodal evaluation.

The MARS2 2025 Challenge on Multimodal Reasoning is an ICCV 2025 workshop challenge organized to evaluate multimodal LLMs and related systems on forms of reasoning that are explicitly framed as more real-world and specialized than standard benchmark tasks. Its design centers on two tailored datasets—Lens and AdsQA—and three tracks: Visual Grounding in Real-world Scenarios (VG-RS), Visual Question Answering with Spatial Awareness (VQA-SA), and Visual Reasoning in Creative Advertisement Videos (VR-Ads). The challenge is positioned as a response to the view that many established multimodal evaluations no longer adequately stress spatial reasoning, cross-task synergy, or implicit abstract interpretation, especially in domains such as advertisement video understanding (Xu et al., 17 Sep 2025).

1. Objectives and intellectual framing

MARS2 2025 is presented as a benchmark and competition for multimodal reasoning in the era of large models. Its central motivation is that recent progress in LLMs, MLLMs, and “large reasoning models” has made many standard tasks less diagnostic, while more difficult reasoning problems remain insufficiently tested. The challenge therefore emphasizes what the organizers describe as more “System 2”-like multimodal reasoning: slower, structured reasoning over physical scenes, spatial relations, and implicit abstract meaning (Xu et al., 17 Sep 2025).

A defining design choice is the separation between real-world and specialized reasoning. The Lens dataset is intended to support general reasoning in 12 daily scenarios while preserving multiple tasks on the same image distribution. This is meant to expose what the paper calls the synergistic effects among reasoning tasks, since both grounding and question answering operate over the same underlying scenes. The AdsQA dataset, by contrast, targets creative advertisement videos, where answers depend on persuasion strategies, emotion, target audience, theme, and narrative logic rather than only literal visual content. This suggests that the challenge is not merely benchmarking perception quality, but probing whether multimodal systems can connect visual evidence to non-literal communicative intent (Xu et al., 17 Sep 2025).

The challenge also situates itself against earlier visual grounding, spatial reasoning, VQA, and video reasoning benchmarks by arguing that many prior resources are fragmented across tasks, rely on synthetic or outdated data, or underemphasize abstract semantics. In that sense, MARS2 2025 functions as both a leaderboard and a benchmark-design intervention: it proposes a unified testbed for grounded real-world reasoning and a separate testbed for domain-specific abstract reasoning (Xu et al., 17 Sep 2025).

2. Datasets and track structure

The challenge consists of three tracks, all described as open-ended QA-style tasks, although the output format differs across tracks. The following table summarizes the official structure.

Track	Dataset	Official output and metric
VG-RS	Lens	Bounding box; [email protected]
VQA-SA	Lens	Open-ended textual answer; LLM-judge accuracy
VR-Ads	AdsQA	Open-ended textual answer; LLM-assisted semantic scoring

Lens is introduced as multi-level evaluation of multimodal reasoning. It contains 3.4K images, 60K+ human-authored questions, 8 tasks, and 12 daily scenarios, with examples such as described object counting, region-wise OCR, and scene knowledge inference. The images were manually collected from social media, and the paper highlights temporal freshness as an anti-contamination measure: 53% of Lens samples were posted after January 2025, while the appendix notes that over 80% were collected after September 2024. The report states that Lens was released as a test set for Tracks #1 and #2, but does not specify the exact train/val/test split counts in the challenge report (Xu et al., 17 Sep 2025).

Track VG-RS uses Lens for visual grounding. The input is an image plus a referring query, and the output is a bounding box. The emphasis is not only on localization but on scene perception, spatial reference resolution, small objects, visually cluttered scenes, and complex referring expressions. Track VQA-SA also uses Lens, but asks for an open-ended textual answer to an image-question pair. It is designed to evaluate spatial reasoning, commonsense reasoning, and counterfactual reasoning on the same image distribution used for grounding. The shared dataset is central to the challenge’s attempt to study cross-task interaction rather than isolated task performance (Xu et al., 17 Sep 2025).

AdsQA is an advertisement video question answering dataset described as, to the authors’ knowledge, the first video QA benchmark for the advertisement domain and the first ad benchmark for LLMs. The full dataset contains 1,544 advertisement videos, 10,962 clips, and 22.7 total hours. Because video QA is computationally expensive, the challenge subset for VR-Ads uses 555 videos and over 3,000 questions. The five open-ended QA task types are visual concept understanding, emotion recognition, theme and core message extraction, persuasion strategy mining, and potential audience modeling. The challenge report gives full-dataset and challenge-subset statistics, but again does not specify a formal train/val/test breakdown (Xu et al., 17 Sep 2025).

A notable methodological distinction between the two datasets is semantic explicitness. Lens largely tests physically grounded reasoning in daily scenes. AdsQA tests what the report describes as implicit, non-physical, and abstract understanding. This suggests that MARS2 2025 deliberately spans both grounded scene understanding and higher-level communicative interpretation (Xu et al., 17 Sep 2025).

3. Evaluation protocol

The official evaluation platform for all tracks is EvalAI, with higher scores interpreted as better performance. Each track uses a different metric family, reflecting its output format and reasoning target (Xu et al., 17 Sep 2025).

For VG-RS, the metric is [email protected]. A prediction is counted as correct if the predicted bounding box has IoU > 0.5 with the ground-truth box. This follows standard visual grounding practice. The challenge report emphasizes that Lens is difficult even for strong generalist MLLMs: among benchmarked models, Qwen2.5-VL-32B reached only 48.47 [email protected] as the strongest organizer baseline, indicating that the task remains far from saturation (Xu et al., 17 Sep 2025).

For VQA-SA, evaluation uses GLM4-flash as an automatic judge. For each predicted answer and human annotation pair, the judge generates multiple candidate judgments and determines correctness via majority voting. The challenge report does not specify the exact number of judge samples or the exact evaluation prompt. This omission is important because the strongest participant systems exploited answer-formatting tactics tailored to LLM judging, including deliberately ambiguous answers and multiple grammatical variants designed to reduce false negatives under automatic evaluation. A plausible implication is that the evaluation protocol itself shaped system behavior in nontrivial ways (Xu et al., 17 Sep 2025).

For VR-Ads, scoring follows AdsQA’s evaluation style using gpt-4o-2024-08-06 for text-similarity assistance together with organizer-defined inclusion and exclusion rules. Scores are discrete: 1.0 for a fully satisfactory answer, 0.5 for a partially matched answer, and 0.0 otherwise. The inclusion rule requires covering as many elements of the ground truth as possible; the exclusion rule sets the score to 0 if the generated answer contains elements not in the ground truth and not inferable from the provided meta-information. This metric reflects the semantic rather than exact-match nature of advertisement reasoning (Xu et al., 17 Sep 2025).

The report also documents difficulty in human terms. On AdsQA, human evaluation on 200 randomly sampled QA pairs using five non-expert evaluators yielded 71.4% average accuracy, compared with 60.7% for Gemini 2.5 Pro on the same benchmark. This establishes a persistent human-model gap on the most abstract track (Xu et al., 17 Sep 2025).

4. Participation, baselines, and leaderboard outcomes

The challenge ran from June 1 to August 6, 2025, in two stages: Stage #1 (01/06/2025 - 02/08/2025) with a public leaderboard and around 900 submissions, and Stage #2 (03/08/2025 - 06/08/2025) after public leaderboard closure with around 300 submissions. The final participation statistics were 76 registered teams, 1200+ submissions, and 40+ valid submissions included in the ranking lists. The organizers state that they imposed no additional restrictions: both open-source and commercial models were allowed, and there was no model-size restriction (Xu et al., 17 Sep 2025).

A major feature of the challenge is the scale of organizer benchmarking. The committee evaluated 40+ baselines spanning generalist MLLMs, task-specific specialist models, open-source and closed-source systems, and model scales from 3B to 72B. Generalist baselines included Qwen2.5-VL, InternVL3, GLM-4.1V-Thinking, GPT-4o, Gemini 2.5 Pro, MiniCPM-o, Deepseek-VL2, and Gemma3. Task-specific baselines for grounding included TransVG, VLTVG, CLIP-VG, MMCA, EEVG, SimVG, and Grounding DINO. Hybrid grounding-generative systems included Groma-7B, Mova-7B, Ferret-7B / 13B, LLM-wrapper, and VLM-R1-3B (Xu et al., 17 Sep 2025).

The strongest organizer baselines were:

• VG-RS: Qwen2.5-VL-32B, 48.47 [email protected]
• VQA-SA: Qwen2.5-VL-32B, 54.10
• VR-Ads: Qwen2.5-VL-7B, 48.04

The final leaderboard best scores were substantially higher:

• VG-RS: 66.70
• VQA-SA: 79.03
• VR-Ads: 56.35

These gaps show different levels of headroom. The VQA-SA track exhibited the largest organizer-baseline to winner gap, from 54.10 to 79.03, suggesting that targeted adaptation can dramatically improve spatial reasoning. By contrast, VR-Ads remained difficult even for the winning system, whose 56.35 score remained below the AdsQA human average of 71.4% (Xu et al., 17 Sep 2025).

The top teams by track were as follows. For VG-RS, the top three were ActiveAlphaAgent (66.70), Star_s (64.83), and Location depends on guessing (64.30). For VQA-SA, the top three were Echoch (79.03), Tele_AI (72.60), and ActiveAlphaAgent (69.72). For VR-Ads, the top three were gogogo_truefaler (56.35), HNU-VPAI (54.62), and ActiveAlphaAgent (53.13) (Xu et al., 17 Sep 2025).

5. Methods and dominant solution patterns

The top-performing systems reveal a coherent methodological picture. Across tracks, successful teams repeatedly used synthetic data generation, supervised fine-tuning, RL-based alignment, prompt engineering, and ensembling. The report states that the dominant RL algorithm was GRPO, inspired by DeepSeek-style reasoning training. At the same time, the organizers caution that many reward functions optimize only task metrics such as IoU, which may be suboptimal and may damage generality or trustworthiness (Xu et al., 17 Sep 2025).

In VG-RS, the winning system, VG-SMART from ActiveAlphaAgent, was built on Qwen2.5-VL and used a multi-stage post-training recipe: domain-specific synthetic data generation, filtering with a custom SNR score, supervised fine-tuning, reinforcement learning with an IoU-centric reward, and distillation from 72B to 7B. The second-place system, DCM-VG from Star_s, is notable because it was training-free and combined Qwen2.5-VL as a generalist bounding-box generator with Grounding DINO as a specialist verifier and refiner. This is the clearest example in the report of generalist–specialist collaboration outperforming either class alone (Xu et al., 17 Sep 2025).

In VQA-SA, the winning system, RSVT from Echoch, used an ensemble of InternVL3-14B, InternVL3-38B, and Qwen2.5-VL-32B, trained with a bilingual cold-start corpus, public spatial reasoning benchmarks, synthetic spatial QA generation, viewpoint transformation augmentation, and self-consistency generation, followed by majority voting. The second-place system, SMART from Tele_AI, used synthetic 3D-rendered spatial scenes in Blender, geometric QA generation, location-aware reasoning chains, cold-start fine-tuning, RL with DAPO, and multiple reasoning trajectories scored by VisualPRM-8B. A particularly interesting empirical observation came from ActiveAlphaAgent, whose STAGES system found that including all questions associated with the same image in the prompt improved accuracy by 2–7 points, especially for stronger reasoning models. This directly supports the challenge’s motivation around cross-question and cross-task synergy (Xu et al., 17 Sep 2025).

In VR-Ads, the winning system, Hi-CoT from gogogo_truefaler, used Qwen2.5-VL-72B with a hierarchical chain-of-thought pipeline: global understanding through audio-visual integration, shot-level segmentation and reasoning, hierarchical combination of global and local evidence, and final answer generation. The implementation explicitly used FFmpeg for audio extraction, Whisper Large-v3 for ASR, and PySceneDetect for scene segmentation. The second-place system, TFA from HNU-VPAI, was a training-free prompting approach whose authors reported that Qwen2.5-VL-72B outperformed GPT-4o and Qwen2.5-Omni, and that audio in Qwen2.5-Omni did not improve performance, likely because much advertisement audio is music rather than semantically informative speech. This suggests that for creative advertisement reasoning, careful structuring of narrative evidence may matter more than simple multimodal accumulation (Xu et al., 17 Sep 2025).

6. Findings, limitations, and significance

The report’s central conclusion is that current multimodal models remain far from robust multimodal reasoning in both grounded daily scenarios and specialized abstract domains. Even after task-specific adaptation, VG-RS remains below 70% at the leaderboard top, and VR-Ads remains substantially below human performance. The paper repeatedly emphasizes qualitative failure modes: confusion between similar objects or materials, weak physical commonsense, viewpoint bias, overlooked subtle visual details, failure to connect those details to theme, and hallucinated or biased interpretations in hard advertisement cases (Xu et al., 17 Sep 2025).

The challenge also identifies several broader design lessons. First, real-world grounding and spatial reasoning remain hard, despite strong frontier MLLMs. Second, specialized advertisement reasoning is even harder, because it requires theme extraction, persuasion analysis, emotion recognition, audience modeling, and implicit communicative reasoning. Third, hybrid systems that combine generalist MLLMs with specialist modules can outperform monolithic approaches, especially in grounding. Fourth, prompting and context aggregation remain major performance levers: multi-question prompts, bilingual prompting, viewpoint stabilization, structured formatting, and carefully chosen resolution/FPS settings all mattered in top systems (Xu et al., 17 Sep 2025).

Important limitations are explicit. The challenge report does not provide full annotation workflows or inter-annotator agreement details for Lens or AdsQA. It also leaves some evaluation internals underspecified, such as the exact prompt and sample count for GLM4-flash judging in VQA-SA. In practical terms, this means that the benchmark is rich and large-scale, but not fully documented at the annotation-protocol level within the challenge paper itself. A plausible implication is that follow-up dataset papers are necessary for complete methodological scrutiny (Xu et al., 17 Sep 2025).

The organizers present the released materials as a durable research resource: Lens and AdsQA, 40+ baseline implementations/results, 15+ participant methods, and leaderboard infrastructure are available through the challenge website and GitHub organization. In this respect, MARS2 2025 is not only a single competition cycle but a benchmark suite for studying the limitations of current MLLMs under grounded real-world reasoning, spatial awareness, and abstract video interpretation. Its broader significance lies in showing that multimodal reasoning remains unsolved precisely where tasks demand cross-task consistency, physically grounded inference, and non-literal semantic interpretation (Xu et al., 17 Sep 2025).