---
title: 'MARS2 2025: Multimodal Reasoning Benchmark'
url: https://www.emergentmind.com/topics/mars2-2025-challenge
type: topic
---

# MARS2 2025: Multimodal Reasoning Benchmark

The **MARS2 2025 Challenge on Multimodal Reasoning** is an ICCV 2025 workshop challenge organized to evaluate multimodal large language models and related systems on forms of reasoning that are explicitly framed as more **real-world** and **specialized** than standard benchmark tasks. Its design centers on two tailored datasets—**Lens** and **AdsQA**—and three tracks: **Visual Grounding in Real-world Scenarios (VG-RS)**, **Visual Question Answering with Spatial Awareness (VQA-SA)**, and **Visual Reasoning in Creative Advertisement Videos (VR-Ads)**. The challenge is positioned as a response to the view that many established multimodal evaluations no longer adequately stress spatial reasoning, cross-task synergy, or implicit abstract interpretation, especially in domains such as advertisement video understanding [2509.14142].

## 1. Objectives and intellectual framing

MARS2 2025 is presented as a benchmark and competition for multimodal reasoning in the era of large models. Its central motivation is that recent progress in LLMs, MLLMs, and “large reasoning models” has made many standard tasks less diagnostic, while more difficult reasoning problems remain insufficiently tested. The challenge therefore emphasizes what the organizers describe as more “System 2”-like multimodal reasoning: slower, structured reasoning over physical scenes, spatial relations, and implicit abstract meaning [2509.14142].

A defining design choice is the separation between **real-world** and **specialized** reasoning. The **Lens** dataset is intended to support general reasoning in 12 daily scenarios while preserving multiple tasks on the same image distribution. This is meant to expose what the paper calls the **synergistic effects among reasoning tasks**, since both grounding and question answering operate over the same underlying scenes. The **AdsQA** dataset, by contrast, targets creative advertisement videos, where answers depend on persuasion strategies, emotion, target audience, theme, and narrative logic rather than only literal visual content. This suggests that the challenge is not merely benchmarking perception quality, but probing whether multimodal systems can connect visual evidence to non-literal communicative intent [2509.14142].

The challenge also situates itself against earlier visual grounding, spatial reasoning, VQA, and video reasoning benchmarks by arguing that many prior resources are fragmented across tasks, rely on synthetic or outdated data, or underemphasize abstract semantics. In that sense, MARS2 2025 functions as both a leaderboard and a benchmark-design intervention: it proposes a unified testbed for grounded real-world reasoning and a separate testbed for domain-specific abstract reasoning [2509.14142].

## 2. Datasets and track structure

The challenge consists of three tracks, all described as open-ended QA-style tasks, although the output format differs across tracks. The following table summarizes the official structure.

| Track | Dataset | Official output and metric |
|---|---|---|
| VG-RS | Lens | Bounding box; **Acc.@0.5** |
| VQA-SA | Lens | Open-ended textual answer; **LLM-judge accuracy** |
| VR-Ads | AdsQA | Open-ended textual answer; **LLM-assisted semantic scoring** |

**Lens** is introduced as **multi-level evaluation of multimodal reasoning**. It contains **3.4K images**, **60K+ human-authored questions**, **8 tasks**, and **12 daily scenarios**, with examples such as described object counting, region-wise OCR, and scene knowledge inference. The images were **manually collected from social media**, and the paper highlights temporal freshness as an anti-contamination measure: **53% of Lens samples were posted after January 2025**, while the appendix notes that over 80% were collected after September 2024. The report states that Lens was released as a **test set** for Tracks #1 and #2, but does **not specify the exact train/val/test split counts** in the challenge report [2509.14142].

Track **VG-RS** uses Lens for visual grounding. The input is an image plus a referring query, and the output is a bounding box. The emphasis is not only on localization but on scene perception, spatial reference resolution, small objects, visually cluttered scenes, and complex referring expressions. Track **VQA-SA** also uses Lens, but asks for an open-ended textual answer to an image-question pair. It is designed to evaluate **spatial reasoning**, **commonsense reasoning**, and **counterfactual reasoning** on the same image distribution used for grounding. The shared dataset is central to the challenge’s attempt to study cross-task interaction rather than isolated task performance [2509.14142].

**AdsQA** is an advertisement video question answering dataset described as, to the authors’ knowledge, the **first video QA benchmark for the advertisement domain** and the **first ad benchmark for LLMs**. The full dataset contains **1,544 advertisement videos**, **10,962 clips**, and **22.7 total hours**. Because video QA is computationally expensive, the challenge subset for **VR-Ads** uses **555 videos** and **over 3,000 questions**. The five open-ended QA task types are visual concept understanding, emotion recognition, theme and core message extraction, persuasion strategy mining, and potential audience modeling. The challenge report gives full-dataset and challenge-subset statistics, but again does **not specify a formal train/val/test breakdown** [2509.14142].

A notable methodological distinction between the two datasets is semantic explicitness. Lens largely tests physically grounded reasoning in daily scenes. AdsQA tests what the report describes as **implicit**, **non-physical**, and **abstract** understanding. This suggests that MARS2 2025 deliberately spans both grounded scene understanding and higher-level communicative interpretation [2509.14142].

## 3. Evaluation protocol

The official evaluation platform for all tracks is **EvalAI**, with higher scores interpreted as better performance. Each track uses a different metric family, reflecting its output format and reasoning target [2509.14142].

For **VG-RS**, the metric is **Acc.@0.5**. A prediction is counted as correct if the predicted bounding box has **IoU > 0.5** with the ground-truth box. This follows standard visual grounding practice. The challenge report emphasizes that Lens is difficult even for strong generalist MLLMs: among benchmarked models, **Qwen2.5-VL-32B** reached only **48.47 Acc.@0.5** as the strongest organizer baseline, indicating that the task remains far from saturation [2509.14142].

For **VQA-SA**, evaluation uses **GLM4-flash** as an automatic judge. For each predicted answer and human annotation pair, the judge generates multiple candidate judgments and determines correctness via **majority voting**. The challenge report does not specify the exact number of judge samples or the exact evaluation prompt. This omission is important because the strongest participant systems exploited answer-formatting tactics tailored to LLM judging, including deliberately ambiguous answers and multiple grammatical variants designed to reduce false negatives under automatic evaluation. A plausible implication is that the evaluation protocol itself shaped system behavior in nontrivial ways [2509.14142].

For **VR-Ads**, scoring follows AdsQA’s evaluation style using **gpt-4o-2024-08-06** for text-similarity assistance together with organizer-defined inclusion and exclusion rules. Scores are discrete: **1.0** for a fully satisfactory answer, **0.5** for a partially matched answer, and **0.0** otherwise. The inclusion rule requires covering as many elements of the ground truth as possible; the exclusion rule sets the score to **0** if the generated answer contains elements not in the ground truth and not inferable from the provided meta-information. This metric reflects the semantic rather than exact-match nature of advertisement reasoning [2509.14142].

The report also documents difficulty in human terms. On **AdsQA**, human evaluation on **200 randomly sampled QA pairs** using five non-expert evaluators yielded **71.4% average accuracy**, compared with **60.7%** for **Gemini 2.5 Pro** on the same benchmark. This establishes a persistent human-model gap on the most abstract track [2509.14142].

## 4. Participation, baselines, and leaderboard outcomes

The challenge ran from **June 1 to August 6, 2025**, in two stages: **Stage #1 (01/06/2025 - 02/08/2025)** with a public leaderboard and around **900 submissions**, and **Stage #2 (03/08/2025 - 06/08/2025)** after public leaderboard closure with around **300 submissions**. The final participation statistics were **76 registered teams**, **1200+ submissions**, and **40+ valid submissions** included in the ranking lists. The organizers state that they imposed **no additional restrictions**: both open-source and commercial models were allowed, and there was **no model-size restriction** [2509.14142].

A major feature of the challenge is the scale of organizer benchmarking. The committee evaluated **40+ baselines** spanning generalist MLLMs, task-specific specialist models, open-source and closed-source systems, and model scales from **3B to 72B**. Generalist baselines included **Qwen2.5-VL**, **InternVL3**, **GLM-4.1V-Thinking**, **GPT-4o**, **Gemini 2.5 Pro**, **MiniCPM-o**, **Deepseek-VL2**, and **Gemma3**. Task-specific baselines for grounding included **TransVG**, **VLTVG**, **CLIP-VG**, **MMCA**, **EEVG**, **SimVG**, and **Grounding DINO**. Hybrid grounding-generative systems included **Groma-7B**, **Mova-7B**, **Ferret-7B / 13B**, **LLM-wrapper**, and **VLM-R1-3B** [2509.14142].

The strongest organizer baselines were:

- **VG-RS**: **Qwen2.5-VL-32B**, **48.47 Acc.@0.5**
- **VQA-SA**: **Qwen2.5-VL-32B**, **54.10**
- **VR-Ads**: **Qwen2.5-VL-7B**, **48.04**

The final leaderboard best scores were substantially higher:

- **VG-RS**: **66.70**
- **VQA-SA**: **79.03**
- **VR-Ads**: **56.35**

These gaps show different levels of headroom. The **VQA-SA** track exhibited the largest organizer-baseline to winner gap, from **54.10** to **79.03**, suggesting that targeted adaptation can dramatically improve spatial reasoning. By contrast, **VR-Ads** remained difficult even for the winning system, whose **56.35** score remained below the AdsQA human average of **71.4%** [2509.14142].

The top teams by track were as follows. For **VG-RS**, the top three were **ActiveAlphaAgent** (**66.70**), **Star_s** (**64.83**), and **Location depends on guessing** (**64.30**). For **VQA-SA**, the top three were **Echoch** (**79.03**), **Tele_AI** (**72.60**), and **ActiveAlphaAgent** (**69.72**). For **VR-Ads**, the top three were **gogogo_truefaler** (**56.35**), **HNU-VPAI** (**54.62**), and **ActiveAlphaAgent** (**53.13**) [2509.14142].

## 5. Methods and dominant solution patterns

The top-performing systems reveal a coherent methodological picture. Across tracks, successful teams repeatedly used **synthetic data generation**, **supervised fine-tuning**, **RL-based alignment**, **prompt engineering**, and **ensembling**. The report states that the dominant RL algorithm was **GRPO**, inspired by DeepSeek-style reasoning training. At the same time, the organizers caution that many reward functions optimize only task metrics such as IoU, which may be **suboptimal** and may damage generality or trustworthiness [2509.14142].

In **VG-RS**, the winning system, **VG-SMART** from **ActiveAlphaAgent**, was built on **Qwen2.5-VL** and used a multi-stage post-training recipe: domain-specific synthetic data generation, filtering with a custom **SNR** score, supervised fine-tuning, reinforcement learning with an IoU-centric reward, and distillation from **72B** to **7B**. The second-place system, **DCM-VG** from **Star_s**, is notable because it was **training-free** and combined **Qwen2.5-VL** as a generalist bounding-box generator with **Grounding DINO** as a specialist verifier and refiner. This is the clearest example in the report of **generalist–specialist collaboration** outperforming either class alone [2509.14142].

In **VQA-SA**, the winning system, **RSVT** from **Echoch**, used an ensemble of **InternVL3-14B**, **InternVL3-38B**, and **Qwen2.5-VL-32B**, trained with a bilingual cold-start corpus, public spatial reasoning benchmarks, synthetic spatial QA generation, viewpoint transformation augmentation, and self-consistency generation, followed by **majority voting**. The second-place system, **SMART** from **Tele_AI**, used synthetic **3D-rendered spatial scenes in Blender**, geometric QA generation, location-aware reasoning chains, cold-start fine-tuning, RL with **DAPO**, and multiple reasoning trajectories scored by **VisualPRM-8B**. A particularly interesting empirical observation came from **ActiveAlphaAgent**, whose **STAGES** system found that including **all questions associated with the same image** in the prompt improved accuracy by **2–7 points**, especially for stronger reasoning models. This directly supports the challenge’s motivation around cross-question and cross-task synergy [2509.14142].

In **VR-Ads**, the winning system, **Hi-CoT** from **gogogo_truefaler**, used **Qwen2.5-VL-72B** with a hierarchical chain-of-thought pipeline: global understanding through audio-visual integration, shot-level segmentation and reasoning, hierarchical combination of global and local evidence, and final answer generation. The implementation explicitly used **FFmpeg** for audio extraction, **Whisper Large-v3** for ASR, and **PySceneDetect** for scene segmentation. The second-place system, **TFA** from **HNU-VPAI**, was a training-free prompting approach whose authors reported that **Qwen2.5-VL-72B** outperformed **GPT-4o** and **Qwen2.5-Omni**, and that audio in **Qwen2.5-Omni** did **not** improve performance, likely because much advertisement audio is music rather than semantically informative speech. This suggests that for creative advertisement reasoning, careful structuring of narrative evidence may matter more than simple multimodal accumulation [2509.14142].

## 6. Findings, limitations, and significance

The report’s central conclusion is that current multimodal models remain far from robust multimodal reasoning in both grounded daily scenarios and specialized abstract domains. Even after task-specific adaptation, **VG-RS** remains below **70%** at the leaderboard top, and **VR-Ads** remains substantially below human performance. The paper repeatedly emphasizes qualitative failure modes: confusion between similar objects or materials, weak physical commonsense, viewpoint bias, overlooked subtle visual details, failure to connect those details to theme, and hallucinated or biased interpretations in hard advertisement cases [2509.14142].

The challenge also identifies several broader design lessons. First, **real-world grounding and spatial reasoning remain hard**, despite strong frontier MLLMs. Second, **specialized advertisement reasoning is even harder**, because it requires theme extraction, persuasion analysis, emotion recognition, audience modeling, and implicit communicative reasoning. Third, **hybrid systems** that combine generalist MLLMs with specialist modules can outperform monolithic approaches, especially in grounding. Fourth, **prompting and context aggregation** remain major performance levers: multi-question prompts, bilingual prompting, viewpoint stabilization, structured formatting, and carefully chosen resolution/FPS settings all mattered in top systems [2509.14142].

Important limitations are explicit. The challenge report does **not** provide full annotation workflows or inter-annotator agreement details for Lens or AdsQA. It also leaves some evaluation internals underspecified, such as the exact prompt and sample count for **GLM4-flash** judging in VQA-SA. In practical terms, this means that the benchmark is rich and large-scale, but not fully documented at the annotation-protocol level within the challenge paper itself. A plausible implication is that follow-up dataset papers are necessary for complete methodological scrutiny [2509.14142].

The organizers present the released materials as a durable research resource: **Lens** and **AdsQA**, **40+ baseline implementations/results**, **15+ participant methods**, and leaderboard infrastructure are available through the challenge website and GitHub organization. In this respect, MARS2 2025 is not only a single competition cycle but a benchmark suite for studying the limitations of current MLLMs under grounded real-world reasoning, spatial awareness, and abstract video interpretation. Its broader significance lies in showing that multimodal reasoning remains unsolved precisely where tasks demand cross-task consistency, physically grounded inference, and non-literal semantic interpretation [2509.14142].

Source: https://www.emergentmind.com/topics/mars2-2025-challenge