---
title: Meteorology Multimodal Large Model (MMLM)
url: https://www.emergentmind.com/topics/meteorology-multimodal-large-model-mmlm
type: topic
---

# Meteorology Multimodal Large Model (MMLM)

A Meteorology Multimodal Large Model (MMLM) is an emerging class of meteorology-specialized large models that combine heterogeneous weather modalities—such as gridded meteorological fields, radar or satellite imagery, station time series, and natural language—to perform forecasting, warning generation, expert question answering, and forecast interpretation. In the current literature, the term is used explicitly for systems such as Weather-R1, MeteorPred, and WeatherSyn, while several adjacent works supply architectural or data components for the broader MMLM agenda. At the same time, the literature sharply distinguishes true multimodal systems from unimodal LLM-style weather foundation models: LM-Weather is explicitly “still a numeric time-series model, not yet a multimodal satellite–radar–text system,” and ClimateLLM is “unimodal” despite its LLM backbone [2601.14044, 2508.06859, 2605.07522, 2405.20348, 2502.11059].

## 1. Conceptual scope and boundaries

Within meteorology research, MMLM denotes more than simply applying a transformer or LLM to weather data. The defining property is cross-modal integration. Weather-R1 frames the goal as building a model that can understand “domain-specific weather maps and satellite imagery,” answer “structured meteorological questions that require reasoning,” and provide “logically faithful chains of thought” [2601.14044]. MeteorPred defines an MMLM as a severe-weather-oriented large multimodal model that “directly ingests 4D meteorological inputs” and generates textual warnings and answers [2508.06859]. WeatherSyn positions a meteorology-focused MLLM around the Weather Forecasting Report task, where the model receives meteorological visuals plus an instruction and generates a structured multi-day forecast report [2605.07522].

The boundary of the concept is therefore operational as well as architectural. Some systems are multimodal in the strict sense because they fuse image-like meteorological products with text, or 4D meteorological tensors with language. Others are better read as precursor branches. LM-Weather contributes the “sequence / on-device / personalization layer” toward a future MMLM, but it only handles multivariate temporal sequences from ground stations [2405.20348]. ClimateLLM contributes a frequency-aware gridded forecasting backbone, yet “currently operates on gridded fields only” [2502.11059]. Met$^2$Net treats meteorological variables as distinct modalities in latent space, but this is multimodality in the multi-variable sense rather than in the radar–satellite–text sense [2507.17189].

This layered usage has become a stable pattern across the literature. One branch emphasizes multimodal reasoning and reporting, another emphasizes multimodal severe-weather understanding and warning generation, and a third develops foundation-model components—spectral tokenization, cross-modal prompting, low-rank adaptation, or station-centric attention—that can be incorporated into fuller MMLMs later [2409.19058, 2504.07394, 2603.27135, 2604.15377, 2506.07616, 2508.12291].

## 2. Modalities, representations, and task families

Current work spans a wide range of modality combinations and output targets.

| System | Modalities | Primary output |
|---|---|---|
| Weather-R1 | Weather maps, satellite IR imagery, text | Multiple-choice meteorological reasoning |
| MeteorPred | 4D ERA5 fields, text | Severe-weather QA and warning generation |
| WeatherSyn | ERA5 variable heat-map images, text | Multi-day forecast reports |
| RadarQA | Radar/VIL frames or sequences, text | Quality ratings and assessment reports |
| M3R | NEXRAD radar imagery, PWS time series | Point rainfall nowcasting |
| FuXi-Air | Meteorological forecasts, emissions, pollutant stations | 72-hour air-quality forecasts |
| CLLMate | ERA5 raster data, textual event data | Weather and climate event narratives |
| ClimateBench-M | ERA5, NOAA storm events, HLS satellite imagery | Forecasting, alerts, crop segmentation |

The most explicit MMLM systems are broad in both modality and task definition. Weather-R1 uses seven imaging modalities in WeatherQA—Rain, Phenom, Max Temp, Min Temp, 500hPa, 850hPa, and Land—paired with natural-language questions and multiple-choice options [2601.14044]. MeteorPred uses a 12-hour window of ERA5 over China with \(C = 5 \times 37 = 185\) channels, corresponding to five variables across 37 pressure levels, and supports MC-main, MC-sub, T/F, RSW, and NSW tasks [2508.06859]. WeatherSyn uses 12 single-level ERA5 variables rendered as city-centered heat-map images and conditions generation on aspect-controlled textual instructions for four forecast days [2605.07522].

Other systems widen the modality space in different directions. M3R combines an 8-frame radar sequence over a \(100 \times 100\) km domain with 20 personal weather station variables aligned at 15-minute resolution [2604.15377]. FuXi-Air integrates FuXi-2.0 meteorological forecasts or ERA5, CAMS-GLOB-ANT emission inventories, and hourly station pollutant observations for six pollutants [2506.07616]. CLLMate aligns meteorological raster data from ERA5 with textual event data derived from environmental news, thereby reformulating weather and climate event forecasting as an open-vocabulary multimodal generation problem [2409.19058]. ClimateBench-M aligns ERA5 county-level time series, NOAA Storm Events thunderstorm labels, and NASA HLS multi-spectral satellite imagery on a county-based spatiotemporal scheme [2504.07394].

A separate but increasingly important modality pairing is text with meteorological time series. MTransformer operates on five surface variables—Temp, Press, Humid, Wind, Precip—paired with expert-level captions in MeteoCap-3B, giving a text-to-time-series generation setting rather than image-grounded reasoning [2603.27135]. LM-Weather is again narrower: it models heterogeneous on-device meteorological sequences such as temperature, humidity, wind, and precipitation, and treats trend, seasonal, residual components, plus temporal meta-features, as structured sub-modalities within a numeric time-series pipeline [2405.20348].

These task families expand the function of weather models beyond numerical prediction. Contemporary MMLMs are used to issue severe weather warnings, generate synoptic forecast reports, judge forecast quality, answer expert exam-style questions, and translate meteorological fields into event narratives [2508.06859, 2605.07522, 2508.12291, 2604.24645, 2409.19058].

## 3. Architectural patterns

Despite differences in input type, several architectural motifs recur. One is the use of a strong pretrained multimodal or language backbone with lightweight meteorology-specific adaptation. WeatherSyn fine-tunes the vision encoder, merger, and LLM of Qwen3-VL-8B in supervised and rejection-sampling stages, then freezes the vision tower during DPO [2605.07522]. MeteorPred wraps backbones such as Qwen2.5-VL-7B-Instruct, LLaVA-NeXT-Video-7B, Video-LLaVA-7B, and InternVL3-8B with three plug-and-play modules: Dynamic Time-Gated Fusion (DTGF), Text-Driven Gaussian Spatial Masking (TGS), and Text-Driven Channel Attention (TGCA) [2508.06859]. CLLMate follows a LLaVA-style design with a CLIP ViT-L/14 visual encoder, an MLP adapter, and a Llama3-8B language backbone [2409.19058].

A second motif is explicit structure-aware tokenization or prompting. ClimateLLM converts normalized gridded fields into the frequency domain with a 2D FFT, processes them with a frequency Mixture-of-Experts, and injects temporal-level and variable-level meta-fusion prompts into a GPT-2–style backbone [2502.11059]. MTransformer maps text embeddings into multi-band spectral priors through a Spectral Prompt Generator and applies Spectral-Prompt Cross-Attention inside a latent diffusion transformer [2603.27135]. LM-Weather decomposes each weather sequence into trend, seasonal, and residual components, applies component-specific token, position, and temporal embeddings in a Task Adapter, and uses LoRA in query and value projections of a frozen PLM backbone [2405.20348].

A third motif is cross-modal attention anchored on a physically meaningful query source. M3R uses weather station time series as queries and radar representations as keys and values, so that local station states selectively attend to precipitation signatures in the surrounding radar field [2604.15377]. FuXi-Air first applies self-attention across monitoring stations, then uses cross-attention from station latent states into a ResNet-encoded field of meteorology and emissions [2506.07616]. Met$^2$Net generalizes the same intuition to multi-variable forecasting by assigning separate encoders and decoders to each variable, stacking their latents, and applying variable self-attention inside a Translator that models inter-variable interactions [2507.17189].

A fourth motif is parameter-efficient or deployment-aware adaptation. LM-Weather communicates only low-rank LoRA matrices in federated rounds, with 0.38M communicated parameters per round and 10.38M trainable parameters on device [2405.20348]. MeteorPred also uses LoRA on attention weights and the added adaptive modules [2508.06859]. This suggests a persistent design direction: frozen or mostly frozen backbones, modality- or task-specific front ends, and low-rank or adapter-based updates.

## 4. Datasets, supervision, and benchmark construction

The development of MMLMs has been tightly coupled to new meteorology-specific datasets and benchmarks.

| Benchmark or dataset | Aligned modalities | Noted scale |
|---|---|---|
| WeatherQA | Weather maps/imagery, text, MCQA | 15,400 entries |
| MP-Bench | 4D meteorological data, text captions/warnings | 421,363 pairs |
| MeteoCap-3B | Station time series, expert captions | \(3.15\times10^9+\) observations, ≈105M pairs |
| CLLMate | ERA5 raster data, environmental news text | 26,156 aligned articles |
| ClimateBench-M | ERA5, NOAA events, HLS imagery | 238 counties, 45 ERA5 features, 3,138 chips |
| K-MetBench | Text, expert charts/diagrams, gold rationales | 1,774 unique questions |
| RQA-70K | Radar forecast/observation imagery, text | 70K QA pairs |

WeatherQA is built from real weather analysis products and expert-designed tasks; a random 5% sample was manually validated by two meteorological experts, both reporting \(>95\%\) approval [2601.14044]. MP-Bench pairs ERA5-derived 4D fields with China Meteorological Administration warnings and sampled “normal weather” negatives, creating the first large-scale temporal multimodal dataset for severe weather event prediction [2508.06859]. WSInstruct aligns ERA5 image sets with segmented Area Forecast Discussion synopses and aspect-controlled prompts, while its automated claim extraction reaches 94% extraction F1 against human-validated references [2605.07522]. K-MetBench derives from 25 sessions of the Korean Meteorological Engineer written exam and adds 141 expert-verified rationales to expose gaps in reasoning validity, multimodal chart reading, and Korean locality [2604.24645].

Several works also contribute new annotation methodologies. MeteoCap-3B is created through the Multi-agent Collaborative Captioning pipeline: Multi-Tool Reasoning, Semantic Grounding, Multi-LLM Ensemble Generation, Global Reflection and Self-Correction, followed by human expert quality checks. The reported human QC pass rate is 94.2%, and the MACC pipeline reduces hallucination rate to 3.8% while achieving 4.82 ± 0.15 physical consistency on a 1–5 scale [2603.27135]. RadarQA constructs RQA-70K using a hybrid pipeline that combines expert labeling for 17 perception-based attributes, automated heuristics for 20 metric-based attributes, and GPT-4o for attribute-informed response generation [2508.12291]. CLLMate builds a knowledge graph from environmental news with 6,219 nodes and 19,197 directed edges, then aligns daily event descriptions with ERA5 raster inputs [2409.19058].

Supervision regimes have become correspondingly specialized. Weather-R1 introduces Logically Consistent Reinforcement Fine-Tuning, adding a reward \(R_{LoCo}\) that fires only when the final answer is both well formatted and logically supported by the reasoning process; the total reward is \(r = 0.1 R_{Format} + 0.3 R_{LoCo} + 0.6 R_{Acc}\) [2601.14044]. WeatherSyn supplements SFT with rejection-sampling fine-tuning, where 40 candidate reports per sample are filtered by claim-level correctness and lexical diversity, followed by DPO on preference pairs [2605.07522]. RadarQA stages SFT, GRPO-based RL for structured rating tasks, and a final post-training phase [2508.12291].

## 5. Empirical capabilities and observed gaps

Explicit MMLM systems have reported substantial gains over general-purpose baselines. On WeatherQA, Weather-R1-7B reaches 52.9% overall, improving by 9.8 percentage points over the original Qwen2.5-VL-7B and surpassing Qwen2.5-VL-32B at 52.0%; under LoCo-RFT, Self-Contra falls from 33.23% to 1.82% on WeatherQA and from 20.02% to 2.47% on ScienceQA [2601.14044]. On MP-Bench, the best open-source Qwen2.5-VL-based MMLM reaches 72.37 on MC-main, 58.19 on MC-sub, 87.13 on T/F, 67.46 on RSW, and 2.1 on NSW, compared with a simplified GPT-4o baseline at 11.92, 6.51, 0.19, 14.03, and 0.1 respectively [2508.06859]. WeatherSyn-DPO attains 0.44 BLEU-1, 0.32 ROUGE-L, and 0.25 METEOR on the Weather Forecasting Report task, while also achieving the highest factual-consistency Top-1 rate of 0.33 in LLM and expert ranking [2605.07522]. RadarQA reaches 61.51% overall accuracy on frame rating and 66.17% on sequence rating, with GPT-4 assessment scores of 6.87 and 6.58, outperforming both open-source and API-based general MLLMs [2508.12291].

Adjacent foundation-model and multimodal precursor systems also show that architecture matters even before full multimodality is present. LM-Weather is best in 29 of 32 forecasting settings and 30 of 32 imputation settings, while communicating only 0.38M parameters per round in federated personalization [2405.20348]. ClimateLLM reports 144-hour ACC values of 0.89 for \(z\), 0.94 for \(t\), 0.96 for \(t2m\), and 0.52 for \(u10\), while reducing total training time from 17.6 h in ClimODE to 0.22 h [2502.11059]. M3R achieves RMSE of 2.87–3.28 mm/hr, MAE of 0.33–0.36 mm/hr, and markedly higher CSI than diffusion and time-series baselines [2604.15377]. FuXi-Air completes 72-hour forecasts for six pollutants across multiple monitoring sites within 25–30 seconds and reports RMSE reductions over WRF–CMAQ ranging from 53.95% for \(O_3\) versus SA07 to 67.20% for \(SO_2\) versus CB6 [2506.07616].

At the same time, the literature documents persistent failure modes. K-MetBench reports an average drop of −18.55% from text questions to multimodal chart questions across 55 models, a pronounced reasoning gap where models hallucinate logic despite correct predictions, and a locality gap in which Korean models outperform significantly larger global models on Korean-specific items [2604.24645]. Weather-R1 identifies a reasoning faithfulness gap in which standard RFT can induce self-contradictory reasoning, and RadarQA shows that traditional scalar verification metrics such as CSI, POD, FAR, DISTS, and LPIPS remain substantially less aligned with expert judgment than a domain-tuned MLLM [2601.14044, 2508.12291].

## 6. Limitations, controversies, and future directions

A central limitation is that much of the current literature remains only partially multimodal. LM-Weather and ClimateLLM are strong foundation-model precursors, but they are explicitly not full multimodal weather systems [2405.20348, 2502.11059]. WeatherSyn uses reanalysis or NWP-style maps as images, yet does not incorporate radar, satellite, or station time series directly [2605.07522]. Weather-R1 is a multimodal reasoning VLM, but its current benchmark centers on static imagery and multiple-choice reasoning rather than open-ended operational forecast production [2601.14044]. MeteorPred directly ingests 4D ERA5 fields, but its current deployment domain is China, its warning text is grounded in one national warning system, and its NSW score remains low at about 2.1/5 even for the best model [2508.06859].

A second limitation concerns physical grounding and uncertainty. ClimateLLM is deterministic and explicitly lacks probabilistic outputs or uncertainty quantification [2502.11059]. MeteorPred, M3R, and FuXi-Air all note the need for additional multi-source data, broader spatial coverage, or more explicit physical constraints [2508.06859, 2604.15377, 2506.07616]. Weather-R1 and K-MetBench show that reasoning quality cannot be inferred from answer accuracy alone, which makes explanation faithfulness a genuine safety issue rather than a stylistic preference [2601.14044, 2604.24645]. RadarQA likewise indicates that expert-like forecast assessment requires domain-specific descriptive ability, not only numerical verification [2508.12291].

The trajectory of the field is nevertheless relatively coherent. Multiple papers explicitly propose adding text, image, and time-series modalities into shared backbones, integrating physics-informed constraints, extending from regional to larger-scale coverage, and evaluating under few-shot, zero-shot, and out-of-distribution regimes [2405.20348, 2502.11059, 2508.06859, 2603.27135, 2504.07394]. A plausible synthesis is that future MMLMs will combine modality-specific encoders, shared multimodal transformers, adapter or low-rank specialization, expert-verified reasoning supervision, and benchmark suites that test not only forecast skill but also chart literacy, locality, and logical faithfulness. Current work already supplies most of these ingredients, but not yet in a single unified system.

Source: https://www.emergentmind.com/topics/meteorology-multimodal-large-model-mmlm