ACM Multimedia 2025 Challenge

• ACM Multimedia 2025 Grand Challenge is a suite of benchmark competitions targeting event-enriched image analysis, clinical multimedia retrieval, and sequential micro-expression analysis.
• The challenge emphasizes multimodal reasoning that integrates temporal context, language grounding, and application-specific evaluations beyond surface recognition.
• Robust evaluation methods, including public/private test phases and weighted metric aggregation, ensure reproducibility and drive advancements in context-aware multimedia AI.

The ACM Multimedia 2025 Grand Challenge denotes a set of benchmark competitions organized at, or in conjunction with, ACM Multimedia 2025 that target multimodal understanding in settings where surface-level recognition is insufficient. In the materials represented here, three 2025 benchmarks are especially prominent: the Event-Enriched Image Analysis (EVENTA) Grand Challenge, which centers event-level image retrieval and captioning; ENTRep, which integrates fine-grained ENT endoscopy classification with image and text retrieval under bilingual clinical supervision; and MEGC2025, which frames facial micro-expression analysis as both sequential spot-then-recognize processing and visual question answering. Taken together, these challenges emphasize context, temporality, retrieval, language grounding, and deployment-oriented evaluation rather than isolated recognition accuracy alone (Tran et al., 26 Aug 2025, Nguyen et al., 6 Aug 2025, Fan et al., 18 Jun 2025).

1. Scope, lineage, and challenge portfolio

ACM Multimedia grand challenges have increasingly shifted from narrow perceptual recognition toward multimodal reasoning over context, semantics, and application constraints. Earlier ACM Multimedia Grand Challenge papers on detecting cheapfakes defined a closely related transition: the target problem was not pixel tampering but out-of-context misuse of real news images, where authentic media are paired with misleading captions and conventional forensic cues may be absent. Those papers explicitly described out-of-context misuse as harder than fake media because the image or video may be authentic and only the narrative context is wrong (2207.14534, Dang-Nguyen et al., 2023).

Within the 2025 cycle, the portfolio represented here spans event understanding, medical multimedia, and affective analysis. EVENTA is presented as the first large-scale benchmark for event-level multimodal understanding and is hosted at ACM Multimedia 2025. ENTRep is the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis. MEGC2025 is the 8th Facial Micro-Expression Grand Challenge and is organized in conjunction with ACM Multimedia 2025 (Tran et al., 26 Aug 2025, Nguyen et al., 6 Aug 2025, Fan et al., 18 Jun 2025).

Challenge	Core tasks	Defining emphasis
EVENTA	Event-Enriched Image Retrieval and Captioning; Event-Based Image Retrieval	Contextual, temporal, and semantic event understanding
ENTRep	Fine-grained image classification; image-to-image retrieval; text-to-image retrieval	Bilingual clinical supervision and fine-grained ENT reasoning
MEGC2025	ME Spot-Then-Recognize; ME Visual Question Answering	Sequential ME analysis and MLLM/LVLM-based reasoning

This distribution suggests a common 2025 benchmark logic: multimedia systems are expected to align perception with narrative, diagnosis, or explanation, rather than merely identify visible content.

2. EVENTA and the move to event-level multimodal understanding

EVENTA is framed as a direct response to the limitations of traditional captioning and retrieval systems, which largely focus on people, objects, and scenes while often failing to explain the broader event behind an image. Its target is event-level multimodal understanding: recovering the who, when, where, what, and why behind an image, including event context, significance, and narrative meaning that cannot be directly inferred from pixels alone (Tran et al., 26 Aug 2025).

The benchmark is built on OpenEvents V1, described as a large-scale event-driven corpus assembled from more than a decade of reporting by CNN and The Guardian and spanning 2011–2022. It contains over 200,000 news articles, more than 400,000 images, and more than 30,000 annotated image–event caption pairs, organized into training, public test, and private test splits. The paper further notes that the image queries in the Track 1 Private Set were heavily augmented to simulate realistic conditions (Tran et al., 26 Aug 2025).

EVENTA defines two tracks. Track 1, Event-Enriched Image Retrieval and Captioning, gives participants an image and requires them to retrieve relevant evidence from a curated external article database and then generate captions that integrate that evidence with visual information. The paper explicitly positions this track around retrieval-augmented generation, with the goal of producing descriptions that include entity names, attributes, temporal and spatial cues, event outcomes, and other details that are not visibly obvious. Track 2, Event-Based Image Retrieval, takes a realistic event-related caption as input and requires retrieval of the corresponding images from the database; the difficulty lies in the fact that the queries are long, event-rich, and semantically nuanced rather than short keyword-style captions (Tran et al., 26 Aug 2025).

The evaluation protocol uses a Public Test phase for development and tuning and a Private Test phase for final ranking. Participants may not annotate the test sets, may only use public external datasets and pretrained models, must avoid private datasets and commercial APIs or tools, and are required to release their source code publicly on GitHub and submit a detailed paper before the deadline. Final scores are based on performance on the Private Test set (Tran et al., 26 Aug 2025).

For both tracks, EVENTA uses a weighted harmonic-mean-style aggregation:

$$Overall\ Score = r \cdot \frac{\sum_i w_i}{\left(\sum_i \frac{w_i}{score_i + \epsilon}\right)},$$

where $\epsilon = 10^{-5}$, $w_i$ are the metric weights, $score_i$ are the individual metric values, and $r$ is the number of valid responses in the submission. For Track 1, the metrics are AP, Recall@1, Recall@10, CLIPScore, and CIDEr, with weights

$$[w_{AP}, w_{R@1}, w_{R@10}, w_{CLIPScore}, w_{CIDEr}] = [0.1, 0.2, 0.2, 0.3, 0.2].$$

For Track 2, the metrics are mAP, MRR, Recall@1, Recall@5, and Recall@10, with weights

$$[w_{mAP}, w_{MRR}, w_{R@1}, w_{R@5}, w_{R@10}] = [0.3, 0.2, 0.2, 0.15, 0.15].$$

The organizers state that this aggregation rewards systems that perform consistently across all metrics rather than excelling in only one (Tran et al., 26 Aug 2025).

The challenge attracted 45 registered teams from six countries. In Track 1, Cerebro achieved an overall score of 0.550, followed by SodaBread at 0.547 and Re:zero Slavery at 0.451. In Track 2, NoResources scored 0.577, followed by 23Trinitrotoluen at 0.572 and LastSong at 0.563. The reported results show that retrieval performance could become extremely strong, while event-rich captioning remained harder; the paper notes that retrieving the right contextual evidence appears more tractable than turning that evidence into fluent, factual, event-rich prose (Tran et al., 26 Aug 2025).

3. ENTRep and clinically grounded multimedia reasoning

ENTRep extends the ACM Multimedia 2025 grand challenge format into clinical imaging. Its central claim is that automated analysis of ENT endoscopy should not stop at classification: clinicians also require reliable retrieval of similar cases, both visually and through concise textual descriptions. ENTRep is therefore defined as a benchmark that jointly evaluates fine-grained ENT classification, image-to-image retrieval, and text-to-image retrieval under bilingual clinical supervision (Nguyen et al., 6 Aug 2025).

The dataset is collected from Thong Nhat Hospital in Ho Chi Minh City, Vietnam, under routine ENT clinical conditions. Each image is annotated by clinical experts with anatomical region or classification and condition type in $\{\text{normal}, \text{abnormal}\}$. When available, the image is also paired with a Vietnamese clinical description and an English translation. The paper gives a JSON-style example containing Path, Classification, Type, Description, and DescriptionEN, illustrating the coupling of image-level anatomy with clinically meaningful text (Nguyen et al., 6 Aug 2025).

ENT endoscopy is presented as technically difficult because image appearance depends on endoscope type, illumination, and operator technique; diagnostically important findings are subtle, localized, and unevenly distributed; and the task requires fine-grained distinctions such as laterality and vocal-fold state. The seven anatomical region classes for classification are Ear Right, Ear Left, Nose Right, Nose Left, Throat, VC-open, and VC-closed (Nguyen et al., 6 Aug 2025).

The benchmark is partitioned into training, public test, and private test sets, with server-side scoring used to ensure consistency and fairness. For classification, the paper reports that nose-right accounts for 25.2% of the class distribution, nose-left for 22.5%, throat for 6.3%, and the remaining classes roughly 11–13% each. The public and private classification test sets contain 645 and 646 images, respectively. For image-to-image retrieval, the training set contains 141 image pairs over 141 images, with 140 bidirectional links; the public and private test sets each contain 139 pairs over 139 unique images, with 136 bidirectional links. For text-to-image retrieval, each subset contains 71 pairs, with average query length 5.42 words (Nguyen et al., 6 Aug 2025).

Task 1 uses Accuracy, Precision, Recall, and F1-score computed with a weighted average by class support:

$$\mathrm{Metric}_{\text{weighted}} = \sum_{c \in \mathcal{C}} \frac{n_c}{N}\,\mathrm{Metric}_c.$$

Tasks 2 and 3 use Recall@1 and Mean Reciprocal Rank:

$$\mathrm{R@1} = \mathbb{1}[r_q=1], \qquad \mathrm{MRR} = \frac{1}{N}\sum_{q=1}^{N}\frac{1}{r_q}.$$

The benchmark does not define a more elaborate ranking loss; emphasis is placed on server-side evaluation and private-test final ranking (Nguyen et al., 6 Aug 2025).

The reported leaderboards are highly competitive. In Task 1 classification, WAS achieved 95.82 across the main metrics on the private test set, followed by Soft Mind_AIO at 95.20. In Task 2 image-to-image retrieval, Soft Mind_AIO achieved 92.09 Recall@1 and 95.70 MRR on the private test set. In Task 3 text-to-image retrieval, SoloL led the private test with 92.64 Recall@1 and 95.81 MRR. The paper notes that text-to-image retrieval shows the largest public/private reordering and interprets this as sensitivity to variation in phrasing, synonymy, and query coverage; it explicitly suggests that improving text normalization and cross-lingual alignment would help (Nguyen et al., 6 Aug 2025).

4. MEGC2025 and micro-expression analysis beyond isolated subtasks

MEGC2025 treats micro-expression analysis as both a sequential recognition problem and a multimodal reasoning problem. The challenge is motivated by two claims: conventional pipelines that separate spotting from recognition are suboptimal for long-duration videos, and multimodal LLMs or large vision-LLMs create new opportunities for micro-expression understanding through natural-language interaction (Fan et al., 18 Jun 2025).

Task 1, ME Spot-Then-Recognize (ME-STR), first requires spotting candidate micro-expression intervals in a long video and then passing only correctly spotted samples to the recognition stage for emotion classification. The benchmark recommends SAMM-LV, CAS(ME)$\epsilon = 10^{-5}$0, 4DME, CAS(ME)$\epsilon = 10^{-5}$1, and SMIC-E-long for training, while imposing no restriction on training data. The official ME-STR test set contains 30 long videos total: 10 from the SAMM Challenge dataset and 20 clips cropped from different videos in CAS(ME)$\epsilon = 10^{-5}$2, with frame rates of 200 fps for SAMM and 30 fps for CAS(ME)$\epsilon = 10^{-5}$3 (Fan et al., 18 Jun 2025).

The baseline for ME-STR is MEAN, a unified Micro-Expression Analysis Network with shared layers and separate spotting and recognition branches. The overall score is the product of spotting and recognition F1 scores:

$\epsilon = 10^{-5}$4

This formulation makes the pipeline highly sensitive to spotting quality. On the MEGC2025 ME-STR test set, the reported STRS values are 0.0062 for SAMM and 0.0086 for CAS(ME)$\epsilon = 10^{-5}$5, with many false positives and false negatives in spotting. The paper explicitly concludes that temporal localization is the main bottleneck in end-to-end micro-expression analysis (Fan et al., 18 Jun 2025).

Task 2, ME Visual Question Answering (ME-VQA), reformulates micro-expression analysis as question answering over a sequence of video frames. The input is a micro-expression clip and a natural-language question, and the model must generate a natural-language answer. The questions can target binary properties, multiclass emotion labels, or compound reasoning over action units and expression class. Recommended training datasets are SAMM, CASME II, SMIC, CAS(ME)$\epsilon = 10^{-5}$6, and 4DME; annotations from SAMM, CASME II, and SMIC were converted into an ME-VQA version dataset. The test set contains 24 clips: 7 from the SAMM Challenge dataset and 17 from CAS(ME)$\epsilon = 10^{-5}$7 (Fan et al., 18 Jun 2025).

The baseline model is Qwen2.5VL-3B, evaluated in zero-shot and LoRA-based fine-tuning settings. Three input variants are tested: video input with equally sampled frames, OAO frames, and OF optical flow between onset and apex. Evaluation covers coarse and fine-grained emotion categories using UF1 and UAR, together with BLEU and ROUGE-1 for answer quality. The paper reports that fine-tuning improves performance over zero-shot in most metrics, that OAO and OF generally outperform plain video inputs for expression classification, and that video-input models can still yield better overall answer quality in BLEU and ROUGE-1 (Fan et al., 18 Jun 2025).

A central technical observation is that classification quality and answer quality do not fully coincide. This suggests that future ME-VQA systems may need to optimize emotion discrimination and language generation jointly rather than treating textual output as a by-product of classification.

5. Benchmark design, scoring logic, and reproducibility regimes

The 2025 challenges represented here do not share a single evaluation schema, but they do share a preference for controlled test access and metric designs that penalize brittle systems. EVENTA uses a Public Test phase and a Private Test phase, with final scores computed on the Private Test set and source code release on GitHub required from participants. ENTRep uses server-side scoring, a public leaderboard for the public test, and final ranking based only on private-test scores. MEGC2025 requires algorithms to run on an unseen test set and submit results to a leaderboard (Tran et al., 26 Aug 2025, Nguyen et al., 6 Aug 2025, Fan et al., 18 Jun 2025).

EVENTA’s weighted harmonic-mean aggregation is notable because it explicitly rewards consistency across retrieval and generation metrics. ENTRep’s use of weighted support for classification and rank-based metrics for retrieval reflects a more conventional benchmark structure, but one adapted to fine-grained clinical classes and bilingual text-image alignment. MEGC2025 adopts a pipeline-sensitive composite score for ME-STR and combines classification-oriented and text-overlap metrics for ME-VQA. In all three cases, the metric design reflects task structure rather than simply reusing standard vision benchmarks (Tran et al., 26 Aug 2025, Nguyen et al., 6 Aug 2025, Fan et al., 18 Jun 2025).

The cheapfake challenge series provides a useful contrast. Those earlier ACM Multimedia challenge papers evaluated multimodal misinformation detection with Accuracy, Precision, Recall, F1-score, and Matthews correlation coefficient as effectiveness metrics, together with latency, number of parameters, and model size as efficiency metrics. The benchmark centered image-caption semantic alignment and contextual consistency rather than event retrieval, clinical retrieval, or language generation. This historical comparison clarifies that ACM Multimedia grand challenges have broadened from classification and semantic consistency checking toward richer narrative and domain-grounded multimodal analysis (Dang-Nguyen et al., 2023, 2207.14534).

6. Research significance, applications, and open directions

EVENTA explicitly positions event-enriched image analysis at the intersection of computer vision, natural language processing, and information retrieval. Its applications are stated to include journalism, media analysis, cultural archiving, accessibility, and digital search. The paper argues that the challenge is a step toward “context-aware, narrative-driven multimedia AI,” and that it can support future work in retrieval-augmented generation, multimodal fusion, and knowledge-grounded captioning (Tran et al., 26 Aug 2025).

ENTRep gives the 2025 grand challenge landscape a clinically grounded counterpart. Its future directions include expanding beyond a single institution, incorporating device metadata, adding short video clips, introducing hierarchical labels, defining soft relevance based on anatomical proximity, reporting calibration and latency, exploring domain-adaptive pretraining, developing anatomy-aware objectives, using lightweight adaptation modules, investigating federated learning, improving bilingual querying via structured ontologies, and including clinician-in-the-loop evaluation. These proposals indicate that benchmark evolution is expected to move from static retrieval tasks toward broader clinical workflow integration (Nguyen et al., 6 Aug 2025).

MEGC2025 contributes a different research agenda: micro-expression analysis should be treated simultaneously as temporal detection and recognition, and as multimodal reasoning and explanation. Its main findings are that spotting remains the hardest part of end-to-end analysis, that separate spotting and recognition are insufficient for realistic deployment, and that MLLM or LVLM approaches are promising but still early, especially for fine-grained emotion understanding and precise temporal localization (Fan et al., 18 Jun 2025).

A broader implication emerges when these 2025 challenges are read alongside the cheapfake challenge lineage. The field is no longer organized primarily around the question of whether a signal has been altered. It