---
title: 'MMReview: Multimodal Peer Review Benchmark'
url: https://www.emergentmind.com/topics/mmreview
type: topic
---

# MMReview: Multimodal Peer Review Benchmark

MMReview is a multidisciplinary, multimodal benchmark specifically designed to evaluate and advance large language models (LLMs) and multimodal language models (MLLMs) for automated academic peer review tasks. It was developed in response to significant limitations in prior datasets and evaluation schemes, particularly a lack of modality coverage beyond text and inadequate simulation of the end-to-end scholarly review pipeline. MMReview provides a comprehensive testbed incorporating diverse disciplines, multimodal manuscript content, and expert-written reviews, and establishes rigorous evaluation on step-wise reasoning, human alignment, and robustness to adversarial manipulation [2508.14146].

## 1. Motivation and Benchmark Scope

MMReview addresses a critical need in the scholarly review community, where rapid growth in academic publications has rendered peer review a major bottleneck due to reviewer workload and the complexity of evaluating rich multimodal content (e.g., figures, tables). Prior LLM-based review studies were limited to AI or computer science domains, focused solely on textual inputs, and employed ad-hoc or narrowly tailored metrics. MMReview is designed to (a) expand coverage across scientific fields, (b) introduce all manuscript modalities commonly found in research papers, and (c) provide a unified task and evaluation suite that probes both outputs and reasoning processes in peer review automation [2508.14146].

The benchmark encompasses four major academic disciplines—Artificial Intelligence, Natural Sciences, Engineering Sciences, and Social Sciences—spanning 17 research subfields. It targets the evaluation of both monomodal LLMs and MLLMs capable of processing combinations of text, figures, tables, and full PDF page images.

## 2. Dataset Composition and Structure

MMReview consists of 240 curated samples selected from an initial pool of over 50,000 papers with open peer reviews (sources include OpenReview, NeurIPS, and Nature Communications). Each sample includes:

- **Textual content**: Abstract and key main-body excerpts.
- **Multimodal elements**: Cropped figures, diagrams, data tables, and full PDF page images.
- **Human reviews**: Expert-written pros/cons, integer scores for soundness and presentation (scales: 1–4), and final binary accept/reject decisions.

The disciplinary composition is summarized below:

| Discipline              | No. of Papers | Subfields Included                                          |
|-------------------------|---------------|-------------------------------------------------------------|
| Artificial Intelligence | 115           | ML, Vision, NLP, RL, GNNs, Signal Proc., AI Apps           |
| Natural Sciences        | 63            | Biology/Medicine, Physics, Chemistry, Earth/Env, Math/Stat  |
| Engineering Sciences    | 38            | Materials, Control, Electronic Info, Energy                 |
| Social Sciences         | 24            | Society, Economics, Finance                                 |

This structure enables evaluation of models in realistic, diverse peer-review scenarios.

## 3. Task Suite and Categories

MMReview defines 13 core tasks organized into four thematic categories, designed to simulate the stages of human peer review:

### a) Step-wise Review Generation
- **Summary (S)**: Manuscript summarization.
- **Strengths Evaluation (SE) / Weaknesses Evaluation (WE)**: Listing merits and limitations along Quality, Clarity, Significance, Originality.
- **Soundness/Presetation Scoring (SS/PS)**: Integer ratings (1–4).

### b) Outcome Formulation
- **Conditional Decision (CD)**: Predicting a 1–10 quality score given human-written pros/cons.
- **Direct Decision (DD)**: Scoring from scratch.
- **Chain-of-Thought Decision (CoD)**: Stepwise reasoning—summary, pros/cons, SS, PS, then score.
- **Meta Decision (MD)**: Simulating an area chair’s accept/reject verdict by synthesizing multiple reviews.

### c) Alignment with Human Preferences
- **Pairwise Rank (PR)**: Selecting which of two papers deserves a higher acceptance tier, reflecting human-like ranking.

### d) Robustness to Adversarial Manipulation
- **Fake Strengths (FS) / Fake Weaknesses (FW)**: Detecting polarity-flipped reviewer comments.
- **Prompt Injection (PI)**: Measuring susceptibility to hidden instructions that can bias score outputs.

All tasks are carefully mapped to real peer review workflow steps, from summary and scoring to meta-level synthesis and adversarial response.

## 4. Evaluation Metrics and Methodology

MMReview employs a mixture of automatic and human-centered metrics, with a focus on both outcome accuracy and alignment with expert human judgments. Key metrics include:

- **Text generation similarity** (S, SE, WE): BARTScore, “LLM-as-Judge” human-likeness scoring.
- **Classification accuracy** (MD, PR): e.g., area chair decisions, pairwise rankings.
- **Score deviation** (SS, PS, CD, DD, CoD, FS, FW, PI): Mean Absolute Error (MAE).
- **Standard Generation and Classification Metrics**:
  - BLEU
  - ROUGE-L (\(\text{ROUGE-L} = \frac{(1+\beta^2)\,R\,P}{R+\beta^2\,P}\), with R = recall, P = precision of the LCS)
  - F1-score (\(2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision}+\text{Recall}}\))
- **Human-alignment**: Likert-scale matching by held-out LLM judge.
- **Adversarial robustness**: Changes in MAE and score deviation under polarities or prompt-injection attacks.

Multimodal input (text + figures/tables or PDF page images) is specifically tested for performance and robustness, and benchmark scripts are provided for systematic evaluation [2508.14146].

## 5. Experimental Results and Empirical Insights

Benchmarking on MMReview covered 16 open-source and 5 closed-source LLM/MLLMs, including both small (<7B parameters) and large (GPT-4o, Claude-4, Gemini-2.5) models. Empirical findings include:

- **Scale effects**: Larger (often closed-source) models achieve the highest accuracy on high-level decisions (MD up to ≈85%, PR above 75%).
- **Mid-size model advantage**: On fine-grained rating tasks (SS, PS), mid-sized LLMs sometimes outperform the largest models (lower MAE), indicating diminishing returns for micro-rating accuracy at extreme scale.
- **Chain-of-Thought benefit**: Chain-of-Thought (CoD) decision prompts consistently reduce MAE compared to direct (DD) scoring, evidencing superior alignment with human review processes.
- **Multimodal robustness**: Including visual modalities reduces vulnerability to prompt injection attacks: forced score increases drop from ≈1.2 (text-only) to ≈0.7 (multimodal).
- **Ongoing limitations**: All models struggle with factual consistency, hallucinations, and susceptibility to adversarial polarity flips, revealing limitations in current LLM/MLLM peer review alignment.

## 6. Limitations and Research Directions

The MMReview benchmark, while the most realistic and complete to date, has explicit limitations:

- **Domain gaps**: Coverage of Humanities and biological fields outside Nature Communications is limited.
- **Modality scope**: Presently focused on text, figures, tables, and static PDF; does not incorporate code snippets or video supplements.
- **Length bias**: Tendency for models to reward longer submissions; absence of length-normalized evaluation.
- **Adversarial vulnerabilities**: Despite improvements, models remain sensitive to prompt injection and data-poisoning attacks.

Suggested extensions include expanding to under-represented disciplines, incorporating new modalities, constructing length-normalized scoring protocols, and developing more sophisticated adversarial detection schemes.

## 7. Significance and Impact

MMReview establishes a critical, standardized foundation for evaluating and comparing LLM/MLLM-based automated peer review systems beyond purely textual or single-stage evaluation [2508.14146]. By:

- Enabling systematic comparison across academic disciplines, modalities, and review tasks.
- Simulating a realistic end-to-end peer review process with stepwise and meta evaluation.
- Requiring robust model behavior in the presence of adversarial manipulation and multimodal complexity.

A plausible implication is that MMReview will accelerate both empirical benchmarking and methodological innovation in automated scholarly peer review, moving the field toward systems that are more transparent, reliable, and aligned with expert human judgment. All data, tasks, prompts, and scripts are openly available, facilitating reproducibility and community advancement.

Source: https://www.emergentmind.com/topics/mmreview