HM-Bench: A Comprehensive Benchmark for Multimodal Large Language Models in Hyperspectral Remote Sensing

Abstract: While multimodal LLMs (MLLMs) have made significant strides in natural image understanding, their ability to perceive and reason over hyperspectral image (HSI) remains underexplored, which is a vital modality in remote sensing. The high dimensionality and intricate spectral-spatial properties of HSI pose unique challenges for models primarily trained on RGB data.To address this gap, we introduce Hyperspectral Multimodal Benchmark (HM-Bench), the first benchmark designed specifically to evaluate MLLMs in HSI understanding. We curate a large-scale dataset of 19,337 question-answer pairs across 13 task categories, ranging from basic perception to spectral reasoning. Given that existing MLLMs are not equipped to process raw hyperspectral cubes natively, we propose a dual-modality evaluation framework that transforms HSI data into two complementary representations: PCA-based composite images and structured textual reports. This approach facilitates a systematic comparison of different representation for model performance. Extensive evaluations on 18 representative MLLMs reveal significant difficulties in handling complex spatial-spectral reasoning tasks. Furthermore, our results demonstrate that visual inputs generally outperform textual inputs, highlighting the importance of grounding in spectral-spatial evidence for effective HSI understanding. Dataset and appendix can be accessed at https://github.com/HuoRiLi-Yu/HM-Bench.

• The paper introduces HM-Bench, the first benchmark to evaluate MLLMs on hyperspectral imagery using a dual-modality (PCA image and structured report) VQA protocol.
• It employs a hierarchical task taxonomy with 13 distinct tasks to assess both basic perception and advanced spectral reasoning, revealing challenging near-random accuracies.
• Empirical results highlight differences in modality performance, with PCA image inputs generally outperforming text reports, underscoring the need for specialized HSI encoders.

HM-Bench: A Rigorous Benchmark for Multimodal LLMs in Hyperspectral Remote Sensing

Motivation and Context

The proliferation of Multimodal LLMs (MLLMs) has stimulated intensive research into general visual-language intelligence. However, their applicability to complex scientific data modalities such as hyperspectral imaging (HSI) remains largely untested. HSI is central to remote sensing, providing dense spectral-spatial signatures for semantic analysis over a wide array of Earth and planetary observation tasks. Existing benchmarks focus on natural images (RGB) or limited remote-sensing scenarios, fundamentally neglecting the challenge of high-dimensional spectral reasoning. The "HM-Bench: A Comprehensive Benchmark for Multimodal LLMs in Hyperspectral Remote Sensing" (2604.08884) proposes the first systematic VQA-style evaluation protocol targeting the hyperspectral domain, confronting MLLMs with tasks that require advanced reasoning over spectral fingerprints beyond what RGB-centric models can offer.

Figure 1: Overview of the advantages of HM-Bench over previous benchmarks and the evaluation paradigm for MLLMs in the hyperspectral domain.

HM-Bench Construction: Dataset, Task Taxonomy, and Representation

Dataset Curation

HM-Bench integrates 20 high-fidelity public HSI datasets spanning UAV, airborne, and satellite (including deep-space/planetary) capture platforms. These datasets exhibit extensive heterogeneity—spatial resolutions from centimeters to kilometers, 102–440 spectral bands over ultraviolet to shortwave/thermal infrared, and an array of scene categories (agrarian, urban, natural, Martian terrain). The complete spectral cubes are used, without reduction to RGB proxies, ensuring the benchmark genuinely examines spectral reasoning.

Figure 2: Statistical overview of HM-Bench, including the distribution of question types across core task scenarios and dataset dimensionality.

Hierarchical Task Taxonomy

HM-Bench implements a three-level task taxonomy, stratifying 13 task types into basic perception and advanced reasoning. The taxonomy includes feature recognition, target quantification, spatial localization, composition interpretation, state evaluation, and temporal change detection. Each is further instantiated with domain-specific reasoning requirements, e.g., spectral unmixing, vegetation health analytics, pollution severity assessment, and bi-temporal change detection. Tasks encompass both closed-form (multiple-choice, counting) and analysis problems, which stress spectral-spatial logic.

Figure 3: Hierarchical task taxonomy illustrating the 13 distinct HSI remote sensing tasks divided between perception and reasoning dimensions.

Dual-Modality Representation

Since current MLLMs are not natively compatible with HSI’s high-dimensional cube inputs, HM-Bench introduces two alternative but aligned representations:

• Image Input: PCA-based composite images. The top-12 principal components are arranged in a multi-panel grid, preserving dominant spectral structures as grayscale maps.
• Report Input: Structured textual reports, systematically describing quantitative spectral-spatial features (band statistics, index calculations, regional variability) in an evidence-based manner, devoid of semantic hallucination.

This dual-modality protocol enables direct comparison of model performance and modality robustness on identical data.

Figure 4: HM-Bench curation/evaluation pipeline, detailing question generation, input transformations (PCA & structured report), and standardized inference for MLLMs.

Evaluation Protocol

Eighteen MLLMs—encompassing generalist commercial systems (GPT, Gemini, Claude, Grok), leading open-source models (Qwen-VL, InternVL, GLM, LLaVA variants, DeepSeek, Kimi, Llama-Nemotron-Nano), and remote-sensing-specialized models (GeoChat, GeoLLaVA)—are evaluated in a strict multiple-choice QA protocol. Models are assessed in zero-shot mode under standardized prompt engineering constraints. Each question is posed under both input modalities, and models select an answer from aligned options; accuracy is the only metric, eliminating subjectivity.

Results and Analysis

Aggregate Performance and Task Difficulty

The highest overall accuracy among all models is 43.08% (InternVL3.5-14B, image input), with most top-tier open-source and proprietary models clustering between 35%–43%. These accuracies are only marginally above random baseline, exposing both the difficulty of the tasks and the inability of current architectures to generalize spectral reasoning. Perception tasks (feature recognition, target quantification, localization) are less challenging, while reasoning tasks—particularly spectral unmixing, vegetation health, and change detection—elicit near-random performance, confirming the hardness of multi-modal, non-RGB scientific inference.

Figure 5: Performance comparison of 5 representative MLLMs across 13 tasks, under both PCA image and report input protocols.

Modality Analysis: Image vs. Report Inputs

Image input, i.e., PCA composites, consistently outperforms text input for most models and tasks, but a moderate subset of numerical or temporal-reasoning tasks (notably target quantification, change detection) show parity or even inversion—where report input is competitive or favorable. This effect is especially pronounced in tasks requiring direct numerical analysis of spectral descriptors.

Figure 6: Task-level comparative analysis highlighting input-modality-dependent performance across models, with blue indicating image superiority and red indicating report advantage.

The necessity of visual grounding is evident: MLLMs fail to exploit the structured textual reports as replacement for detailed spatial-spectral context, reinforcing the importance of spectral-spatial vision components in MLLM design.

Open-Source vs. Closed-Source Dynamics

An empirical contradiction to prevailing trends is observed: several open-source MLLMs (InternVL, Qwen-VL, GLM, GeoLLaVA) occasionally match or exceed the performance of proprietary models. This is traced to three factors: (i) many tasks minimize the advantage of generic world knowledge and privilege visual discrimination, (ii) evaluation restricts output form to strictly multiple-choice, curbing open-ended dialogic strengths, and (iii) PCA composites and reports diverge significantly from natural-image/web-text pretraining distributions, penalizing models optimized for mainstream language/vision tasks. Domain-adaptive vision-language pretraining is essential for non-RGB scientific modalities.

Practical and Theoretical Implications

By formalizing a challenging HSI domain benchmark, HM-Bench sets a new standard for measuring the spectral-spatial cognitive capacity of MLLMs. Empirical results underline current model incapacity for advanced remote sensing reasoning, mandating the development of specialized HSI encoders (capable of directly ingesting spectral cubes), domain-aligned contrastive pretraining, and architectural innovations for spectral-spatial feature fusion. Practically, progress in these directions is a prerequisite for deploying next-generation MLLMs in high-impact Earth/planetary science tasks: environmental monitoring, precision agriculture, planetary composition analysis, and dynamic change supervision.

The benchmark’s modality analysis also provides rigorous evidence for the unsuitability of pure language-based or RGB-adapted models in scientific imaging workflows, contrasting with outcomes commonly reported in general vision-language evaluation.

Future Research Directions

Opportunities for advancing scientific multimodal models include:

• Joint pretraining on richly annotated HSI with explicit spectral reasoning objectives.
• Unified architectures integrating spectral cube encoders with LLM backbones.
• Hybrid decision systems robustly fusing visual and textual modalities at multiple stages.
• Expansion of the benchmark to larger, more diverse, and temporally dynamic datasets, including real-world annotation and inter-annotator agreement metrics for QA scenarios.

Direct ingestion of raw high-dimensional HSI cubes, physiochemical knowledge integration, and principled multimodal grounding are mandatory advancements to close the demonstrated gap.

Conclusion

"HM-Bench" establishes the definitive evaluation matrix for HSI-centric VQA in the MLLM paradigm, exposing both the qualitative and quantitative limitations of contemporary multimodal AI on scientific imagery. The rigorous taxonomy, faithfully preserved spectral information, and dual-modality representation protocol together provide a foundation for reproducible comparison and the systematic advancement of spectral vision-language intelligence. The field’s next leaps will be measured—and hopefully catalyzed—by performance on HM-Bench (2604.08884).