---
title: 'MMESGBench: ESG Multimodal Reasoning Benchmark'
url: https://www.emergentmind.com/topics/mmesgbench
type: topic
---

# MMESGBench: ESG Multimodal Reasoning Benchmark

Searching arXiv for MMESGBench and related ESG QA benchmarks.
MMESGBench is a benchmark dataset for multimodal document understanding and complex reasoning in the environmental, social, and governance (ESG) domain. It is designed to evaluate question answering over real-world ESG documents that are long, structurally diverse, multimodal, and frequently require document-level or cross-page reasoning rather than isolated page interpretation. The benchmark was introduced in “Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks” [2507.18932], which defines MMESGBench as a human-AI collaborative benchmark containing 933 validated question-answer pairs derived from 45 ESG documents spanning seven document types and three major ESG source categories. A naming ambiguity exists in adjacent literature: “ESGBench: A Benchmark for Explainable ESG Question Answering in Corporate Sustainability Reports” [2511.16438] presents a distinct benchmark for explainable ESG question answering over sustainability reports and notes that “MMESGBench” may sometimes be used as a naming variant or mistaken reference in informal discussion. Within the benchmark literature itself, however, MMESGBench most precisely denotes the multimodal, cross-page ESG reasoning benchmark of [2507.18932].

## 1. Definition and scope

MMESGBench evaluates four capabilities in ESG document understanding: multimodal understanding of documents, reasoning across pages, answerability detection, and grounding in text, tables, charts, images, and layout [2507.18932]. Its target documents are not restricted to corporate sustainability reports. Instead, the benchmark covers a broader ESG documentary ecosystem comprising corporate ESG reports, ESG standards and frameworks, and government and international organization documents.

The benchmark is motivated by three properties of ESG materials. First, they exhibit multi-source diversity, drawing from heterogeneous institutional settings and disclosure regimes. Second, they are intrinsically multimodal, combining dense narrative text with tables, charts, figures, images, and layout-dependent cues. Third, they exhibit structural complexity: documents may span dozens to over 2,000 pages, with an average length of 157 pages, making long-context and cross-page reasoning essential [2507.18932].

This framing distinguishes MMESGBench from narrower ESG question-answering settings focused primarily on textual extraction from sustainability reports. A plausible implication is that MMESGBench occupies the intersection of document intelligence, multimodal QA, and ESG-specific reasoning, rather than functioning merely as a finance-domain adaptation of standard text QA.

## 2. Corpus composition and benchmark structure

MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning seven document types and three major ESG source categories [2507.18932]. The question distribution is explicitly reported as 58.5% single-page, 25.6% cross-page, and 15.9% unanswerable. This tripartite structure makes answerability detection a first-class evaluation target rather than an incidental failure mode.

The three major ESG source categories are summarized below.

| Source category | Examples explicitly mentioned | Role in benchmark |
|---|---|---|
| Corporate ESG Reports | annual sustainability reports, CDP climate responses | Real-world corporate disclosures |
| ESG Standards and Frameworks | GRI, TCFD, SASB, GHG Protocol, ISO, UNGC, TNFD | Normative and compliance-oriented documents |
| Government and International Organization Documents | SDGs, IPCC reports, NGFS guidelines | Public-policy and global governance materials |

The benchmark also spans seven document types, although the provided source text does not enumerate all seven labels in one place [2507.18932]. What is explicit is that the collection covers a broad spread of formats and structures across the three major source categories.

Question types are labeled both by reasoning scope and by reasoning form. By scope, the labels are single-page, cross-page, and unanswerable. By content and reasoning form, the benchmark includes factual extraction, analytical/compliance reasoning, quantitative reasoning, visually grounded reasoning, and multi-hop reasoning [2507.18932]. This labeling schema is central to the benchmark’s diagnostic value because it separates failures caused by local extraction from those caused by multimodal synthesis or long-range reasoning.

## 3. Construction pipeline

MMESGBench is constructed through a human-AI collaborative, multi-stage pipeline [2507.18932]. The process begins with document collection: the authors curated 45 representative ESG PDFs from public repositories, with selection criteria emphasizing multimodal richness and document complexity. The documents are retained in PDF format to preserve layout and visual semantics, an important design choice because layout itself is part of the semantic signal under evaluation.

Candidate QA generation then proceeds in three branches: single-page, cross-page, and unanswerable generation. For single-page QA generation, each document page is rendered as an image and processed by Qwen-VL-Max. The model is prompted with a small set of example QA pairs and jointly attends to text, layout, and visual elements. It generates candidate questions in three main forms: factual extraction, analytical/compliance-related reasoning, and quantitative reasoning [2507.18932]. The paper explicitly notes that these questions may be visually grounded, requiring table, chart, or image understanding.

Cross-page QA generation is built on a semantic clustering pipeline. Each page is encoded using PaliGemma-3B; embeddings are projected into a 128-dimensional space; a FAISS-based vector index is built over all page embeddings; nearest-neighbor retrieval identifies semantically similar pages; and clustering groups pages by topic [2507.18932]. Multimodal LLMs then generate questions requiring information from multiple pages, such as aggregation across sections, temporal or metric comparison, and causal or referential reasoning. This suggests that MMESGBench operationalizes cross-page reasoning through latent topical structure rather than through arbitrary page concatenation.

Unanswerable QA generation is handled by prompting the model to create plausible questions that fit the document theme but lack supporting evidence. These items are intended to evaluate answerability detection, hallucination resistance, and robustness of LLMs [2507.18932]. Their explicit inclusion differentiates MMESGBench from many document QA datasets in which abstention behavior is not systematically measured.

After candidate generation, an LLM verification stage evaluates each QA pair for accuracy, completeness, and answerability. The verifier also assigns a difficulty score based on reasoning depth and multimodal complexity. An important quality-control filter removes questions that can be answered confidently without document context: the pipeline first performs no-context inference using Qwen-Max, and if the question can be answered without document evidence, the sample is removed [2507.18932]. This filter aims to suppress generic-world-knowledge items and enforce genuine document grounding.

Flagged or uncertain examples then undergo expert-in-the-loop validation by ESG and AI experts, plus trained annotators, through a retrieval-assisted interface. They check evidence traceability, verify factual consistency, ensure multimodal alignment, and revise or discard weak samples [2507.18932]. In effect, the benchmark’s construction process couples automated scale with targeted human calibration.

## 4. Evidence annotations and multimodal grounding

MMESGBench is explicitly built around layout-aware multimodal semantics. Each QA pair is annotated with fine-grained evidence metadata that records the originating modality as text, table, chart, image, or layout [2507.18932]. The dataset overview further notes that evidence sources include both structured and unstructured elements, with pure text and layout accounting for the largest share of examples.

This evidence structure is methodologically important because it changes what “correctness” means in document QA. In a purely text-centric benchmark, correctness can often be approximated as semantic equivalence between a predicted answer and a reference answer. In MMESGBench, correctness is entangled with modality access and document grounding. A system may fail not because it cannot reason linguistically, but because it cannot interpret chart geometry, recover table structure, or exploit layout-dependent semantics.

The benchmark’s evidence annotations also permit failure decomposition by modality and location. Since performance is reported by evidence modality and by evidence location—single-page, cross-page, and unanswerable—the benchmark supports a more granular diagnosis of whether a system’s weakness lies in retrieval, visual parsing, long-context reasoning, or answerability calibration [2507.18932]. This suggests that MMESGBench functions as both a leaderboard benchmark and an analysis instrument for multimodal ESG reasoning pipelines.

## 5. Evaluation protocol and metrics

MMESGBench uses a three-stage evaluation protocol: free-form response generation, automatic short-form answer extraction, and rule-based score computation [2507.18932]. The two principal metrics are Accuracy, defined as exact match with the reference answer, and generalized macro-F1, which balances performance across answerable and unanswerable questions and accounts for partial matches and abstentions.

Performance is additionally decomposed by evidence modality—text, layout, chart, table, image—and by evidence location—single-page, cross-page, unanswerable [2507.18932]. This decomposition is not merely descriptive; it is part of the benchmark’s core analytical design because ESG documents combine heterogeneous evidence channels that stress different components of a model.

The following table summarizes the benchmark’s reported evaluation structure.

| Aspect | Reported protocol or metric | Purpose |
|---|---|---|
| Response pipeline | free-form response generation | Initial model output |
| Extraction stage | automatic short-form answer extraction | Canonicalization for scoring |
| Score computation | rule-based score computation | Final metric assignment |
| Main metric 1 | Accuracy | Exact match with reference answer |
| Main metric 2 | generalized macro-F1 | Balances answerable/unanswerable and partial matches |
| Breakdowns | evidence modality; evidence location | Diagnostic analysis |

The use of generalized macro-F1 is especially consequential because unanswerable items can otherwise distort evaluation. The paper observes that some text-only models over-predict unanswerable cases, which can inflate accuracy on negatives while hurting macro-F1 [2507.18932]. This is a recurrent issue in abstention-aware QA benchmarks: raw accuracy may obscure pathological calibration strategies.

## 6. Baselines and empirical findings

The paper evaluates 15 models in three groups: text-only LLMs, multimodal LLMs, and RAG pipelines including text retrieval and multimodal retrieval [2507.18932]. The results establish a clear ranking among architectural classes: text-only baselines perform poorly; retrieval materially improves performance; multimodal models outperform text-only models; and multimodal RAG systems perform best overall.

Among text-only baselines, the reported examples are ChatGLM-128k with 14.3 ACC / 9.6 F1, Mistral-Instruct-v0.1 with 22.2 ACC / 14.2 F1, Qwen-14B-Chat with 20.4 ACC / 12.9 F1, and Qwen-Max with 24.5 ACC / 25.1 F1 [2507.18932]. Text RAG substantially improves these numbers: ColBERT + Mistral-Instruct reaches 37.0 ACC / 35.9 F1, and ColBERT + Qwen-Max reaches 41.5 ACC / 40.9 F1.

Among multimodal LLMs, the strongest reported baseline is Qwen-VL-Max with 40.0 ACC / 38.3 F1 [2507.18932]. The best overall results come from multimodal RAG, specifically ColPali + Qwen-VL-Max with 5-page retrieval at 51.8 ACC / 51.3 F1; the 1-page variant is also strong at 48.2 ACC / 47.1 F1.

These results are summarized below.

| Model class | Example system | Reported score |
|---|---|---|
| Text-only LLM | Qwen-Max | 24.5 ACC / 25.1 F1 |
| Text RAG | ColBERT + Qwen-Max | 41.5 ACC / 40.9 F1 |
| Multimodal LLM | Qwen-VL-Max | 40.0 ACC / 38.3 F1 |
| Multimodal RAG | ColPali + Qwen-VL-Max, 5-page retrieval | 51.8 ACC / 51.3 F1 |

The paper highlights that multimodal and retrieval-augmented models substantially outperform text-only baselines, particularly on visually grounded and cross-page tasks [2507.18932]. The strongest gains occur on tasks involving tables, charts, figures, layout-dependent semantics, and cross-page reasoning, where multimodal encoders and retrieval pipelines alleviate evidence-access bottlenecks.

Persistent weaknesses remain in chart-based questions, fine-grained spatial reasoning, numerical interpretation, and layout-intensive reasoning without visual encoders [2507.18932]. The empirical picture is therefore not simply that “multimodality helps,” but that current multimodal systems still exhibit brittle behavior on high-fidelity document understanding tasks that require structural parsing and numerical grounding.

## 7. Relation to ESGBench, limitations, and research significance

MMESGBench should be distinguished from ESGBench [2511.16438]. ESGBench is a benchmark for explainable ESG question answering in corporate sustainability reports, emphasizing evidence-grounded extraction of report facts and numeric KPIs from passages and tables. Its initial release contains 10 companies, 12 PDFs, and 119 QA pairs, with categories Environmental, Social, Governance, Strategy, and Risk [2511.16438]. By contrast, MMESGBench is broader in source coverage, explicitly multimodal, and centered on cross-page reasoning and answerability over long ESG documents [2507.18932]. The two benchmarks are related by domain but differ in task design, evidence structure, and evaluation emphasis. The naming ambiguity noted in [2511.16438] therefore reflects terminological overlap in ESG benchmark discourse, not identity of the datasets.

MMESGBench also has explicit limitations. The benchmark text indicates that the dataset spans seven document types, but the exact seven labels are not fully enumerated in the provided source excerpt [2507.18932]. More substantively, the benchmark’s strongest gains still leave absolute performance modest, with the best system reaching 51.8 ACC / 51.3 F1. This suggests that ESG document intelligence remains far from solved even when multimodal retrieval is available.

The paper’s core significance lies in demonstrating that ESG documents are not well captured by either conventional text-only QA or generic multimodal benchmarks. Their combination of heterogeneous source types, long-range dependencies, and layout-sensitive semantics creates a benchmark regime in which retrieval, visual grounding, answerability calibration, and reasoning all matter simultaneously [2507.18932]. A plausible implication is that MMESGBench can serve as a stress test for document AI systems intended for regulatory analysis, sustainability auditing support, standards interpretation, and other ESG-adjacent workflows where failure modes are often traceable to missing evidence rather than to abstract language reasoning alone.

In that sense, MMESGBench occupies a specific place in the recent benchmark landscape. ESGBench [2511.16438] emphasizes explainable QA over sustainability reports with verbatim evidence and KPI fidelity, whereas MMESGBench [2507.18932] emphasizes multimodal understanding and complex reasoning over a wider ESG document ecology. Taken together, they indicate an emerging benchmark taxonomy within ESG AI: one strand focused on faithful evidence-grounded extraction from corporate disclosures, and another focused on multimodal, cross-page, and answerability-aware reasoning over structurally complex ESG documentation.

Source: https://www.emergentmind.com/topics/mmesgbench