MMESGBench: ESG Multimodal Reasoning Benchmark

• MMESGBench is a benchmark dataset for evaluating multimodal ESG document reasoning with human-AI collaboration, containing 933 validated QA pairs.
• The benchmark employs a multi-stage pipeline combining visual, text, and layout information to assess single-page, cross-page, and unanswerable questions.
• Empirical results reveal multimodal RAG systems outperform text-only baselines, highlighting challenges in fine-grained spatial, numerical, and layout reasoning.

Searching arXiv for MMESGBench and related ESG QA benchmarks. MMESGBench is a benchmark dataset for multimodal document understanding and complex reasoning in the environmental, social, and governance (ESG) domain. It is designed to evaluate question answering over real-world ESG documents that are long, structurally diverse, multimodal, and frequently require document-level or cross-page reasoning rather than isolated page interpretation. The benchmark was introduced in “Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks” (Zhang et al., 25 Jul 2025), which defines MMESGBench as a human-AI collaborative benchmark containing 933 validated question-answer pairs derived from 45 ESG documents spanning seven document types and three major ESG source categories. A naming ambiguity exists in adjacent literature: “ESGBench: A Benchmark for Explainable ESG Question Answering in Corporate Sustainability Reports” (George et al., 20 Nov 2025) presents a distinct benchmark for explainable ESG question answering over sustainability reports and notes that “MMESGBench” may sometimes be used as a naming variant or mistaken reference in informal discussion. Within the benchmark literature itself, however, MMESGBench most precisely denotes the multimodal, cross-page ESG reasoning benchmark of (Zhang et al., 25 Jul 2025).

1. Definition and scope

MMESGBench evaluates four capabilities in ESG document understanding: multimodal understanding of documents, reasoning across pages, answerability detection, and grounding in text, tables, charts, images, and layout (Zhang et al., 25 Jul 2025). Its target documents are not restricted to corporate sustainability reports. Instead, the benchmark covers a broader ESG documentary ecosystem comprising corporate ESG reports, ESG standards and frameworks, and government and international organization documents.

The benchmark is motivated by three properties of ESG materials. First, they exhibit multi-source diversity, drawing from heterogeneous institutional settings and disclosure regimes. Second, they are intrinsically multimodal, combining dense narrative text with tables, charts, figures, images, and layout-dependent cues. Third, they exhibit structural complexity: documents may span dozens to over 2,000 pages, with an average length of 157 pages, making long-context and cross-page reasoning essential (Zhang et al., 25 Jul 2025).

This framing distinguishes MMESGBench from narrower ESG question-answering settings focused primarily on textual extraction from sustainability reports. A plausible implication is that MMESGBench occupies the intersection of document intelligence, multimodal QA, and ESG-specific reasoning, rather than functioning merely as a finance-domain adaptation of standard text QA.

2. Corpus composition and benchmark structure

MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning seven document types and three major ESG source categories (Zhang et al., 25 Jul 2025). The question distribution is explicitly reported as 58.5% single-page, 25.6% cross-page, and 15.9% unanswerable. This tripartite structure makes answerability detection a first-class evaluation target rather than an incidental failure mode.

The three major ESG source categories are summarized below.

Source category	Examples explicitly mentioned	Role in benchmark
Corporate ESG Reports	annual sustainability reports, CDP climate responses	Real-world corporate disclosures
ESG Standards and Frameworks	GRI, TCFD, SASB, GHG Protocol, ISO, UNGC, TNFD	Normative and compliance-oriented documents
Government and International Organization Documents	SDGs, IPCC reports, NGFS guidelines	Public-policy and global governance materials

The benchmark also spans seven document types, although the provided source text does not enumerate all seven labels in one place (Zhang et al., 25 Jul 2025). What is explicit is that the collection covers a broad spread of formats and structures across the three major source categories.

Question types are labeled both by reasoning scope and by reasoning form. By scope, the labels are single-page, cross-page, and unanswerable. By content and reasoning form, the benchmark includes factual extraction, analytical/compliance reasoning, quantitative reasoning, visually grounded reasoning, and multi-hop reasoning (Zhang et al., 25 Jul 2025). This labeling schema is central to the benchmark’s diagnostic value because it separates failures caused by local extraction from those caused by multimodal synthesis or long-range reasoning.

3. Construction pipeline

MMESGBench is constructed through a human-AI collaborative, multi-stage pipeline (Zhang et al., 25 Jul 2025). The process begins with document collection: the authors curated 45 representative ESG PDFs from public repositories, with selection criteria emphasizing multimodal richness and document complexity. The documents are retained in PDF format to preserve layout and visual semantics, an important design choice because layout itself is part of the semantic signal under evaluation.

Candidate QA generation then proceeds in three branches: single-page, cross-page, and unanswerable generation. For single-page QA generation, each document page is rendered as an image and processed by Qwen-VL-Max. The model is prompted with a small set of example QA pairs and jointly attends to text, layout, and visual elements. It generates candidate questions in three main forms: factual extraction, analytical/compliance-related reasoning, and quantitative reasoning (Zhang et al., 25 Jul 2025). The paper explicitly notes that these questions may be visually grounded, requiring table, chart, or image understanding.

Cross-page QA generation is built on a semantic clustering pipeline. Each page is encoded using PaliGemma-3B; embeddings are projected into a 128-dimensional space; a FAISS-based vector index is built over all page embeddings; nearest-neighbor retrieval identifies semantically similar pages; and clustering groups pages by topic (Zhang et al., 25 Jul 2025). Multimodal LLMs then generate questions requiring information from multiple pages, such as aggregation across sections, temporal or metric comparison, and causal or referential reasoning. This suggests that MMESGBench operationalizes cross-page reasoning through latent topical structure rather than through arbitrary page concatenation.

Unanswerable QA generation is handled by prompting the model to create plausible questions that fit the document theme but lack supporting evidence. These items are intended to evaluate answerability detection, hallucination resistance, and robustness of LLMs (Zhang et al., 25 Jul 2025). Their explicit inclusion differentiates MMESGBench from many document QA datasets in which abstention behavior is not systematically measured.

After candidate generation, an LLM verification stage evaluates each QA pair for accuracy, completeness, and answerability. The verifier also assigns a difficulty score based on reasoning depth and multimodal complexity. An important quality-control filter removes questions that can be answered confidently without document context: the pipeline first performs no-context inference using Qwen-Max, and if the question can be answered without document evidence, the sample is removed (Zhang et al., 25 Jul 2025). This filter aims to suppress generic-world-knowledge items and enforce genuine document grounding.

Flagged or uncertain examples then undergo expert-in-the-loop validation by ESG and AI experts, plus trained annotators, through a retrieval-assisted interface. They check evidence traceability, verify factual consistency, ensure multimodal alignment, and revise or discard weak samples (Zhang et al., 25 Jul 2025). In effect, the benchmark’s construction process couples automated scale with targeted human calibration.

4. Evidence annotations and multimodal grounding

MMESGBench is explicitly built around layout-aware multimodal semantics. Each QA pair is annotated with fine-grained evidence metadata that records the originating modality as text, table, chart, image, or layout (Zhang et al., 25 Jul 2025). The dataset overview further notes that evidence sources include both structured and unstructured elements, with pure text and layout accounting for the largest share of examples.

This evidence structure is methodologically important because it changes what “correctness” means in document QA. In a purely text-centric benchmark, correctness can often be approximated as semantic equivalence between a predicted answer and a reference answer. In MMESGBench, correctness is entangled with modality access and document grounding. A system may fail not because it cannot reason linguistically, but because it cannot interpret chart geometry, recover table structure, or exploit layout-dependent semantics.

The benchmark’s evidence annotations also permit failure decomposition by modality and location. Since performance is reported by evidence modality and by evidence location—single-page, cross-page, and unanswerable—the benchmark supports a more granular diagnosis of whether a system’s weakness lies in retrieval, visual parsing, long-context reasoning, or answerability calibration (Zhang et al., 25 Jul 2025). This suggests that MMESGBench functions as both a leaderboard benchmark and an analysis instrument for multimodal ESG reasoning pipelines.

5. Evaluation protocol and metrics

MMESGBench uses a three-stage evaluation protocol: free-form response generation, automatic short-form answer extraction, and rule-based score computation (Zhang et al., 25 Jul 2025). The two principal metrics are Accuracy, defined as exact match with the reference answer, and generalized macro-F1, which balances performance across answerable and unanswerable questions and accounts for partial matches and abstentions.

Performance is additionally decomposed by evidence modality—text, layout, chart, table, image—and by evidence location—single-page, cross-page, unanswerable (Zhang et al., 25 Jul 2025). This decomposition is not merely descriptive; it is part of the benchmark’s core analytical design because ESG documents combine heterogeneous evidence channels that stress different components of a model.

The following table summarizes the benchmark’s reported evaluation structure.

Aspect	Reported protocol or metric	Purpose
Response pipeline	free-form response generation	Initial model output
Extraction stage	automatic short-form answer extraction	Canonicalization for scoring
Score computation	rule-based score computation	Final metric assignment
Main metric 1	Accuracy	Exact match with reference answer
Main metric 2	generalized macro-F1	Balances answerable/unanswerable and partial matches
Breakdowns	evidence modality; evidence location	Diagnostic analysis

The use of generalized macro-F1 is especially consequential because unanswerable items can otherwise distort evaluation. The paper observes that some text-only models over-predict unanswerable cases, which can inflate accuracy on negatives while hurting macro-F1 (Zhang et al., 25 Jul 2025). This is a recurrent issue in abstention-aware QA benchmarks: raw accuracy may obscure pathological calibration strategies.

6. Baselines and empirical findings

The paper evaluates 15 models in three groups: text-only LLMs, multimodal LLMs, and RAG pipelines including text retrieval and multimodal retrieval (Zhang et al., 25 Jul 2025). The results establish a clear ranking among architectural classes: text-only baselines perform poorly; retrieval materially improves performance; multimodal models outperform text-only models; and multimodal RAG systems perform best overall.

Among text-only baselines, the reported examples are ChatGLM-128k with 14.3 ACC / 9.6 F1, Mistral-Instruct-v0.1 with 22.2 ACC / 14.2 F1, Qwen-14B-Chat with 20.4 ACC / 12.9 F1, and Qwen-Max with 24.5 ACC / 25.1 F1 (Zhang et al., 25 Jul 2025). Text RAG substantially improves these numbers: ColBERT + Mistral-Instruct reaches 37.0 ACC / 35.9 F1, and ColBERT + Qwen-Max reaches 41.5 ACC / 40.9 F1.

Among multimodal LLMs, the strongest reported baseline is Qwen-VL-Max with 40.0 ACC / 38.3 F1 (Zhang et al., 25 Jul 2025). The best overall results come from multimodal RAG, specifically ColPali + Qwen-VL-Max with 5-page retrieval at 51.8 ACC / 51.3 F1; the 1-page variant is also strong at 48.2 ACC / 47.1 F1.

These results are summarized below.

Model class	Example system	Reported score
Text-only LLM	Qwen-Max	24.5 ACC / 25.1 F1
Text RAG	ColBERT + Qwen-Max	41.5 ACC / 40.9 F1
Multimodal LLM	Qwen-VL-Max	40.0 ACC / 38.3 F1
Multimodal RAG	ColPali + Qwen-VL-Max, 5-page retrieval	51.8 ACC / 51.3 F1

The paper highlights that multimodal and retrieval-augmented models substantially outperform text-only baselines, particularly on visually grounded and cross-page tasks (Zhang et al., 25 Jul 2025). The strongest gains occur on tasks involving tables, charts, figures, layout-dependent semantics, and cross-page reasoning, where multimodal encoders and retrieval pipelines alleviate evidence-access bottlenecks.

Persistent weaknesses remain in chart-based questions, fine-grained spatial reasoning, numerical interpretation, and layout-intensive reasoning without visual encoders (Zhang et al., 25 Jul 2025). The empirical picture is therefore not simply that “multimodality helps,” but that current multimodal systems still exhibit brittle behavior on high-fidelity document understanding tasks that require structural parsing and numerical grounding.

7. Relation to ESGBench, limitations, and research significance

MMESGBench should be distinguished from ESGBench (George et al., 20 Nov 2025). ESGBench is a benchmark for explainable ESG question answering in corporate sustainability reports, emphasizing evidence-grounded extraction of report facts and numeric KPIs from passages and tables. Its initial release contains 10 companies, 12 PDFs, and 119 QA pairs, with categories Environmental, Social, Governance, Strategy, and Risk (George et al., 20 Nov 2025). By contrast, MMESGBench is broader in source coverage, explicitly multimodal, and centered on cross-page reasoning and answerability over long ESG documents (Zhang et al., 25 Jul 2025). The two benchmarks are related by domain but differ in task design, evidence structure, and evaluation emphasis. The naming ambiguity noted in (George et al., 20 Nov 2025) therefore reflects terminological overlap in ESG benchmark discourse, not identity of the datasets.

MMESGBench also has explicit limitations. The benchmark text indicates that the dataset spans seven document types, but the exact seven labels are not fully enumerated in the provided source excerpt (Zhang et al., 25 Jul 2025). More substantively, the benchmark’s strongest gains still leave absolute performance modest, with the best system reaching 51.8 ACC / 51.3 F1. This suggests that ESG document intelligence remains far from solved even when multimodal retrieval is available.

The paper’s core significance lies in demonstrating that ESG documents are not well captured by either conventional text-only QA or generic multimodal benchmarks. Their combination of heterogeneous source types, long-range dependencies, and layout-sensitive semantics creates a benchmark regime in which retrieval, visual grounding, answerability calibration, and reasoning all matter simultaneously (Zhang et al., 25 Jul 2025). A plausible implication is that MMESGBench can serve as a stress test for document AI systems intended for regulatory analysis, sustainability auditing support, standards interpretation, and other ESG-adjacent workflows where failure modes are often traceable to missing evidence rather than to abstract language reasoning alone.

In that sense, MMESGBench occupies a specific place in the recent benchmark landscape. ESGBench (George et al., 20 Nov 2025) emphasizes explainable QA over sustainability reports with verbatim evidence and KPI fidelity, whereas MMESGBench (Zhang et al., 25 Jul 2025) emphasizes multimodal understanding and complex reasoning over a wider ESG document ecology. Taken together, they indicate an emerging benchmark taxonomy within ESG AI: one strand focused on faithful evidence-grounded extraction from corporate disclosures, and another focused on multimodal, cross-page, and answerability-aware reasoning over structurally complex ESG documentation.