Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

Abstract: Multi-spectral imagery is a valuable input signal for Remote Sensing applications, such as land-use and land-cover classification and environmental monitoring. However, generalist Large Multi-modal Models (LMMs) are typically trained on RGB images, limiting their applicability to the RGB domain. At the same time, training multi-spectral multi-modal models is expensive and produces uniquely specialized models. To address this, we propose a novel training-free approach that introduces multi-spectral data within the inference pipeline of standard RGB-only LMMs, allowing large gains in performance. Our approach leverages the LMMs' understanding of the visual space by adapting non-RGB inputs to that space and injecting domain-specific information and Chain-of-Thought reasoning as instructions. We demonstrate this with the Gemini 2.5 model and observe strong Zero-Shot performance gains on popular Remote Sensing benchmarks. These results highlight the potential for geospatial professionals to leverage powerful generalist models for specialized sensor inputs, benefiting from rich reasoning capabilities grounded in specialized data.

• The paper introduces a novel, inference-time adaptation method for RGB-trained LMMs, enabling them to interpret multi-spectral satellite data through pseudo-images and domain-specific prompts.
• It utilizes multi-step Chain-of-Thought reasoning to iteratively refine classifications, achieving state-of-the-art zero-shot performance on benchmarks like BigEarthNet and EuroSat.
• The study demonstrates a scalable approach that bypasses costly retraining, thereby expanding LMM applications to diverse remote sensing modalities with improved accuracy.

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

Introduction

The paper "Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning" (2604.21032) addresses the critical limitation in the deployment of generalist Large Multi-Modal Models (LMMs) for Remote Sensing tasks: their constraint to RGB-only inputs. This limitation precludes their applicability to multi-sensor satellite imagery where non-RGB bands (NIR, SWIR, derived indices) are key for tasks such as land cover/use classification and environmental monitoring. The proposed method introduces an inference-time, training-free adaptation pipeline that enables RGB-only LMMs to process multi-spectral data using pseudo-imagery and explicit domain-specific prompts. The approach further leverages multi-step Chain-of-Thought (CoT) reasoning to maximize zero-shot inference performance.

Figure 1: Schematic of the proposed adaptation—RGB-trained LMMs interpret previously unfamiliar multi-spectral imagery via prompt engineering and pseudo-images, vastly extending their application space without retraining.

This adaptation strategy is demonstrated on Gemini 2.5, yielding SOTA zero-shot performance on established remote sensing benchmarks without fine-tuning or retraining, thus establishing a highly scalable and robust methodology.

Multi-Spectral Data Integration Beyond RGB

The central innovation of the paper is the transformation of multi-spectral sensor data into a pseudo-image format (false color composites, NDVI, NDWI, multi-band NDMI) compatible with the LMM’s RGB-centric visual pipeline. These synthesized images, alongside detailed textual prompts describing band composition and physical meaning, allow the model to ground these unfamiliar modalities within its pre-trained visual understanding space.

Figure 2: Representative modal inputs derived from multi-spectral satellite sources—showing the range of indices and composites used to encapsulate sensor data as LMM-compatible images.

Unlike domain-specific model retraining or costly multi-modal fine-tuning, this technique is strictly inference-driven, immediately extensible to new sensors and spectral configurations. Critical to this success is the exploitation of language modality: the prompt provides not only explicit band and modality specifications but also semantically disambiguates confusing or overlapping class descriptors, compensating for gaps in vision-language alignment for unseen inputs.

Reasoning Strategies: Informative Prompting, Vocabulary Expansion, and Chain-of-Thought

Three prompting strategies are evaluated:

• Informative Prompting integrates explicit descriptions of generated pseudo-images and band composition;
• Vocabulary Expansion supplements class labels with fine-grained, domain-relevant definitions to resolve semantic ambiguities;
• Chain-of-Thought (CoT) Reasoning adopts a three-stage prompt (propose, verify, conclude) requiring the model to iteratively analyze the input, justify candidate classifications, and select the output according to gathered evidence.

The CoT strategy achieves the most significant improvement, confirming that multi-step explicit reasoning substantially enhances the extraction of salient and discriminative attributes from entangled multi-modal inputs.

Figure 3: Failure cases of the RGB-only model contrasted with precise classification by the multi-spectral, CoT-enabled Gemini—highlighting superior disambiguation of water and forested terrain.

Experimental Results

BigEarthNet

The Gemini-based pipeline achieves an F1 score of 0.523 with CoT and multi-spectral inputs on BigEarthNet (19-class, multi-label), substantially outperforming both RGB-only zero-shot baselines (0.414) and prior SOTA models including remote sensing-adapted LMMs and dedicated multi-spectral architectures. Gains are observed across all metrics (precision, recall), with the most pronounced when combining full spectral data and CoT reasoning.

EuroSat

On EuroSat (multi-class task), the model reaches 72.7% Top-1 accuracy (multi-spectral, CoT), a marked improvement over SOTA zero-shot multi-modal methods and on par with fine-tuned domain-specific models, despite no task-specific adaptation. Results consistently show that every increment of input spectral diversity and reasoning context translates into monotonic accuracy improvement.

Ablation

Comprehensive ablation confirms that both the addition of non-RGB bands and elaborate, context-rich prompts (band descriptions, pseudo-image specifications) are necessary for optimal domain transfer. Removal of either component consistently degrades performance, underlining their complementary roles.

Implications and Future Outlook

This training-free modality adaptation marks a transition in how remote sensing and other sensor-rich domains can exploit industrial-scale LMMs. The method mitigates the high cost and brittleness of sensor-specific model development, repositioning generalist models as viable, adaptable solutions for rapidly evolving, heterogeneous data landscapes.

On a theoretical level, the research demonstrates that LMMs’ latent capacity for compositional visual reasoning is extensible far beyond their training set via robust prompt engineering, particularly when paired with structured, stepwise reasoning frameworks like CoT. This defines a generic pattern for domain adaptation in high-dimensional, multi-modal tasks.

Practically, the approach enables rapid, cost-effective deployment of advanced geospatial analytics pipelines, facilitating integration with modern sensor constellations (e.g., next-gen satellites with novel band architectures), and can potentially be extended to modalities such as LiDAR, thermal, and SAR through similar visual-linguistic interface synthesis.

Challenges remain in translating all forms of high-dimensional sensor data into interpretable pseudo-images and prompts, especially for modalities fundamentally misaligned with vision-oriented representations.

Conclusion

The paper presents a scalable, robust pipeline to unlock multi-spectral inputs for RGB-trained LMMs, eliminating the need for retraining. Through guided pseudo-image construction and advanced Chain-of-Thought reasoning, it achieves state-of-the-art zero-shot classification on core remote sensing benchmarks. The results emphasize the latent extensibility of LMMs and pave the way for prompt-driven, domain-agnostic integration of noncanonical sensors in vision-language architectures (2604.21032).