Meteorology Multimodal Large Model (MMLM)

• Meteorology Multimodal Large Models (MMLMs) are specialized systems that integrate heterogeneous weather modalities—such as gridded fields, radar/satellite imagery, time series, and text—to enhance forecasting and warning generation.
• They employ cross-modal architectures using dynamic fusion, tailored attention mechanisms, and adapter-based updates to jointly process visual, numerical, and natural language data.
• Empirical evaluations demonstrate that MMLMs outperform unimodal counterparts on benchmarks while addressing challenges in physical grounding and uncertainty quantification.

A Meteorology Multimodal Large Model (MMLM) is an emerging class of meteorology-specialized large models that combine heterogeneous weather modalities—such as gridded meteorological fields, radar or satellite imagery, station time series, and natural language—to perform forecasting, warning generation, expert question answering, and forecast interpretation. In the current literature, the term is used explicitly for systems such as Weather-R1, MeteorPred, and WeatherSyn, while several adjacent works supply architectural or data components for the broader MMLM agenda. At the same time, the literature sharply distinguishes true multimodal systems from unimodal LLM-style weather foundation models: LM-Weather is explicitly “still a numeric time-series model, not yet a multimodal satellite–radar–text system,” and ClimateLLM is “unimodal” despite its LLM backbone (Wu et al., 20 Jan 2026, Tang et al., 9 Aug 2025, Zheng et al., 8 May 2026, Chen et al., 2024, Li et al., 16 Feb 2025).

1. Conceptual scope and boundaries

Within meteorology research, MMLM denotes more than simply applying a transformer or LLM to weather data. The defining property is cross-modal integration. Weather-R1 frames the goal as building a model that can understand “domain-specific weather maps and satellite imagery,” answer “structured meteorological questions that require reasoning,” and provide “logically faithful chains of thought” (Wu et al., 20 Jan 2026). MeteorPred defines an MMLM as a severe-weather-oriented large multimodal model that “directly ingests 4D meteorological inputs” and generates textual warnings and answers (Tang et al., 9 Aug 2025). WeatherSyn positions a meteorology-focused MLLM around the Weather Forecasting Report task, where the model receives meteorological visuals plus an instruction and generates a structured multi-day forecast report (Zheng et al., 8 May 2026).

The boundary of the concept is therefore operational as well as architectural. Some systems are multimodal in the strict sense because they fuse image-like meteorological products with text, or 4D meteorological tensors with language. Others are better read as precursor branches. LM-Weather contributes the “sequence / on-device / personalization layer” toward a future MMLM, but it only handles multivariate temporal sequences from ground stations (Chen et al., 2024). ClimateLLM contributes a frequency-aware gridded forecasting backbone, yet “currently operates on gridded fields only” (Li et al., 16 Feb 2025). Met$^2$Net treats meteorological variables as distinct modalities in latent space, but this is multimodality in the multi-variable sense rather than in the radar–satellite–text sense (Li et al., 23 Jul 2025).

This layered usage has become a stable pattern across the literature. One branch emphasizes multimodal reasoning and reporting, another emphasizes multimodal severe-weather understanding and warning generation, and a third develops foundation-model components—spectral tokenization, cross-modal prompting, low-rank adaptation, or station-centric attention—that can be incorporated into fuller MMLMs later (Li et al., 2024, Fu et al., 10 Apr 2025, Zhang, 28 Mar 2026, Panta et al., 15 Apr 2026, Geng et al., 9 Jun 2025, He et al., 17 Aug 2025).

2. Modalities, representations, and task families

Current work spans a wide range of modality combinations and output targets.

System	Modalities	Primary output
Weather-R1	Weather maps, satellite IR imagery, text	Multiple-choice meteorological reasoning
MeteorPred	4D ERA5 fields, text	Severe-weather QA and warning generation
WeatherSyn	ERA5 variable heat-map images, text	Multi-day forecast reports
RadarQA	Radar/VIL frames or sequences, text	Quality ratings and assessment reports
M3R	NEXRAD radar imagery, PWS time series	Point rainfall nowcasting
FuXi-Air	Meteorological forecasts, emissions, pollutant stations	72-hour air-quality forecasts
CLLMate	ERA5 raster data, textual event data	Weather and climate event narratives
ClimateBench-M	ERA5, NOAA storm events, HLS satellite imagery	Forecasting, alerts, crop segmentation

The most explicit MMLM systems are broad in both modality and task definition. Weather-R1 uses seven imaging modalities in WeatherQA—Rain, Phenom, Max Temp, Min Temp, 500hPa, 850hPa, and Land—paired with natural-language questions and multiple-choice options (Wu et al., 20 Jan 2026). MeteorPred uses a 12-hour window of ERA5 over China with $C = 5 \times 37 = 185$ channels, corresponding to five variables across 37 pressure levels, and supports MC-main, MC-sub, T/F, RSW, and NSW tasks (Tang et al., 9 Aug 2025). WeatherSyn uses 12 single-level ERA5 variables rendered as city-centered heat-map images and conditions generation on aspect-controlled textual instructions for four forecast days (Zheng et al., 8 May 2026).

Other systems widen the modality space in different directions. M3R combines an 8-frame radar sequence over a $100 \times 100$ km domain with 20 personal weather station variables aligned at 15-minute resolution (Panta et al., 15 Apr 2026). FuXi-Air integrates FuXi-2.0 meteorological forecasts or ERA5, CAMS-GLOB-ANT emission inventories, and hourly station pollutant observations for six pollutants (Geng et al., 9 Jun 2025). CLLMate aligns meteorological raster data from ERA5 with textual event data derived from environmental news, thereby reformulating weather and climate event forecasting as an open-vocabulary multimodal generation problem (Li et al., 2024). ClimateBench-M aligns ERA5 county-level time series, NOAA Storm Events thunderstorm labels, and NASA HLS multi-spectral satellite imagery on a county-based spatiotemporal scheme (Fu et al., 10 Apr 2025).

A separate but increasingly important modality pairing is text with meteorological time series. MTransformer operates on five surface variables—Temp, Press, Humid, Wind, Precip—paired with expert-level captions in MeteoCap-3B, giving a text-to-time-series generation setting rather than image-grounded reasoning (Zhang, 28 Mar 2026). LM-Weather is again narrower: it models heterogeneous on-device meteorological sequences such as temperature, humidity, wind, and precipitation, and treats trend, seasonal, residual components, plus temporal meta-features, as structured sub-modalities within a numeric time-series pipeline (Chen et al., 2024).

These task families expand the function of weather models beyond numerical prediction. Contemporary MMLMs are used to issue severe weather warnings, generate synoptic forecast reports, judge forecast quality, answer expert exam-style questions, and translate meteorological fields into event narratives (Tang et al., 9 Aug 2025, Zheng et al., 8 May 2026, He et al., 17 Aug 2025, Kim et al., 27 Apr 2026, Li et al., 2024).

3. Architectural patterns

Despite differences in input type, several architectural motifs recur. One is the use of a strong pretrained multimodal or language backbone with lightweight meteorology-specific adaptation. WeatherSyn fine-tunes the vision encoder, merger, and LLM of Qwen3-VL-8B in supervised and rejection-sampling stages, then freezes the vision tower during DPO (Zheng et al., 8 May 2026). MeteorPred wraps backbones such as Qwen2.5-VL-7B-Instruct, LLaVA-NeXT-Video-7B, Video-LLaVA-7B, and InternVL3-8B with three plug-and-play modules: Dynamic Time-Gated Fusion (DTGF), Text-Driven Gaussian Spatial Masking (TGS), and Text-Driven Channel Attention (TGCA) (Tang et al., 9 Aug 2025). CLLMate follows a LLaVA-style design with a CLIP ViT-L/14 visual encoder, an MLP adapter, and a Llama3-8B language backbone (Li et al., 2024).

A second motif is explicit structure-aware tokenization or prompting. ClimateLLM converts normalized gridded fields into the frequency domain with a 2D FFT, processes them with a frequency Mixture-of-Experts, and injects temporal-level and variable-level meta-fusion prompts into a GPT-2–style backbone (Li et al., 16 Feb 2025). MTransformer maps text embeddings into multi-band spectral priors through a Spectral Prompt Generator and applies Spectral-Prompt Cross-Attention inside a latent diffusion transformer (Zhang, 28 Mar 2026). LM-Weather decomposes each weather sequence into trend, seasonal, and residual components, applies component-specific token, position, and temporal embeddings in a Task Adapter, and uses LoRA in query and value projections of a frozen PLM backbone (Chen et al., 2024).

A third motif is cross-modal attention anchored on a physically meaningful query source. M3R uses weather station time series as queries and radar representations as keys and values, so that local station states selectively attend to precipitation signatures in the surrounding radar field (Panta et al., 15 Apr 2026). FuXi-Air first applies self-attention across monitoring stations, then uses cross-attention from station latent states into a ResNet-encoded field of meteorology and emissions (Geng et al., 9 Jun 2025). Met$^2$Net generalizes the same intuition to multi-variable forecasting by assigning separate encoders and decoders to each variable, stacking their latents, and applying variable self-attention inside a Translator that models inter-variable interactions (Li et al., 23 Jul 2025).

A fourth motif is parameter-efficient or deployment-aware adaptation. LM-Weather communicates only low-rank LoRA matrices in federated rounds, with 0.38M communicated parameters per round and 10.38M trainable parameters on device (Chen et al., 2024). MeteorPred also uses LoRA on attention weights and the added adaptive modules (Tang et al., 9 Aug 2025). This suggests a persistent design direction: frozen or mostly frozen backbones, modality- or task-specific front ends, and low-rank or adapter-based updates.

4. Datasets, supervision, and benchmark construction

The development of MMLMs has been tightly coupled to new meteorology-specific datasets and benchmarks.

Benchmark or dataset	Aligned modalities	Noted scale
WeatherQA	Weather maps/imagery, text, MCQA	15,400 entries
MP-Bench	4D meteorological data, text captions/warnings	421,363 pairs
MeteoCap-3B	Station time series, expert captions	$3.15\times10^9+$ observations, ≈105M pairs
CLLMate	ERA5 raster data, environmental news text	26,156 aligned articles
ClimateBench-M	ERA5, NOAA events, HLS imagery	238 counties, 45 ERA5 features, 3,138 chips
K-MetBench	Text, expert charts/diagrams, gold rationales	1,774 unique questions
RQA-70K	Radar forecast/observation imagery, text	70K QA pairs

WeatherQA is built from real weather analysis products and expert-designed tasks; a random 5% sample was manually validated by two meteorological experts, both reporting $>95\%$ approval (Wu et al., 20 Jan 2026). MP-Bench pairs ERA5-derived 4D fields with China Meteorological Administration warnings and sampled “normal weather” negatives, creating the first large-scale temporal multimodal dataset for severe weather event prediction (Tang et al., 9 Aug 2025). WSInstruct aligns ERA5 image sets with segmented Area Forecast Discussion synopses and aspect-controlled prompts, while its automated claim extraction reaches 94% extraction F1 against human-validated references (Zheng et al., 8 May 2026). K-MetBench derives from 25 sessions of the Korean Meteorological Engineer written exam and adds 141 expert-verified rationales to expose gaps in reasoning validity, multimodal chart reading, and Korean locality (Kim et al., 27 Apr 2026).

Several works also contribute new annotation methodologies. MeteoCap-3B is created through the Multi-agent Collaborative Captioning pipeline: Multi-Tool Reasoning, Semantic Grounding, Multi-LLM Ensemble Generation, Global Reflection and Self-Correction, followed by human expert quality checks. The reported human QC pass rate is 94.2%, and the MACC pipeline reduces hallucination rate to 3.8% while achieving 4.82 ± 0.15 physical consistency on a 1–5 scale (Zhang, 28 Mar 2026). RadarQA constructs RQA-70K using a hybrid pipeline that combines expert labeling for 17 perception-based attributes, automated heuristics for 20 metric-based attributes, and GPT-4o for attribute-informed response generation (He et al., 17 Aug 2025). CLLMate builds a knowledge graph from environmental news with 6,219 nodes and 19,197 directed edges, then aligns daily event descriptions with ERA5 raster inputs (Li et al., 2024).

Supervision regimes have become correspondingly specialized. Weather-R1 introduces Logically Consistent Reinforcement Fine-Tuning, adding a reward $R_{LoCo}$ that fires only when the final answer is both well formatted and logically supported by the reasoning process; the total reward is $r = 0.1 R_{Format} + 0.3 R_{LoCo} + 0.6 R_{Acc}$ (Wu et al., 20 Jan 2026). WeatherSyn supplements SFT with rejection-sampling fine-tuning, where 40 candidate reports per sample are filtered by claim-level correctness and lexical diversity, followed by DPO on preference pairs (Zheng et al., 8 May 2026). RadarQA stages SFT, GRPO-based RL for structured rating tasks, and a final post-training phase (He et al., 17 Aug 2025).

5. Empirical capabilities and observed gaps

Explicit MMLM systems have reported substantial gains over general-purpose baselines. On WeatherQA, Weather-R1-7B reaches 52.9% overall, improving by 9.8 percentage points over the original Qwen2.5-VL-7B and surpassing Qwen2.5-VL-32B at 52.0%; under LoCo-RFT, Self-Contra falls from 33.23% to 1.82% on WeatherQA and from 20.02% to 2.47% on ScienceQA (Wu et al., 20 Jan 2026). On MP-Bench, the best open-source Qwen2.5-VL-based MMLM reaches 72.37 on MC-main, 58.19 on MC-sub, 87.13 on T/F, 67.46 on RSW, and 2.1 on NSW, compared with a simplified GPT-4o baseline at 11.92, 6.51, 0.19, 14.03, and 0.1 respectively (Tang et al., 9 Aug 2025). WeatherSyn-DPO attains 0.44 BLEU-1, 0.32 ROUGE-L, and 0.25 METEOR on the Weather Forecasting Report task, while also achieving the highest factual-consistency Top-1 rate of 0.33 in LLM and expert ranking (Zheng et al., 8 May 2026). RadarQA reaches 61.51% overall accuracy on frame rating and 66.17% on sequence rating, with GPT-4 assessment scores of 6.87 and 6.58, outperforming both open-source and API-based general MLLMs (He et al., 17 Aug 2025).

Adjacent foundation-model and multimodal precursor systems also show that architecture matters even before full multimodality is present. LM-Weather is best in 29 of 32 forecasting settings and 30 of 32 imputation settings, while communicating only 0.38M parameters per round in federated personalization (Chen et al., 2024). ClimateLLM reports 144-hour ACC values of 0.89 for $z$, 0.94 for $t$, 0.96 for $C = 5 \times 37 = 185$0, and 0.52 for $C = 5 \times 37 = 185$1, while reducing total training time from 17.6 h in ClimODE to 0.22 h (Li et al., 16 Feb 2025). M3R achieves RMSE of 2.87–3.28 mm/hr, MAE of 0.33–0.36 mm/hr, and markedly higher CSI than diffusion and time-series baselines (Panta et al., 15 Apr 2026). FuXi-Air completes 72-hour forecasts for six pollutants across multiple monitoring sites within 25–30 seconds and reports RMSE reductions over WRF–CMAQ ranging from 53.95% for $C = 5 \times 37 = 185$2 versus SA07 to 67.20% for $C = 5 \times 37 = 185$3 versus CB6 (Geng et al., 9 Jun 2025).

At the same time, the literature documents persistent failure modes. K-MetBench reports an average drop of −18.55% from text questions to multimodal chart questions across 55 models, a pronounced reasoning gap where models hallucinate logic despite correct predictions, and a locality gap in which Korean models outperform significantly larger global models on Korean-specific items (Kim et al., 27 Apr 2026). Weather-R1 identifies a reasoning faithfulness gap in which standard RFT can induce self-contradictory reasoning, and RadarQA shows that traditional scalar verification metrics such as CSI, POD, FAR, DISTS, and LPIPS remain substantially less aligned with expert judgment than a domain-tuned MLLM (Wu et al., 20 Jan 2026, He et al., 17 Aug 2025).

6. Limitations, controversies, and future directions

A central limitation is that much of the current literature remains only partially multimodal. LM-Weather and ClimateLLM are strong foundation-model precursors, but they are explicitly not full multimodal weather systems (Chen et al., 2024, Li et al., 16 Feb 2025). WeatherSyn uses reanalysis or NWP-style maps as images, yet does not incorporate radar, satellite, or station time series directly (Zheng et al., 8 May 2026). Weather-R1 is a multimodal reasoning VLM, but its current benchmark centers on static imagery and multiple-choice reasoning rather than open-ended operational forecast production (Wu et al., 20 Jan 2026). MeteorPred directly ingests 4D ERA5 fields, but its current deployment domain is China, its warning text is grounded in one national warning system, and its NSW score remains low at about 2.1/5 even for the best model (Tang et al., 9 Aug 2025).

A second limitation concerns physical grounding and uncertainty. ClimateLLM is deterministic and explicitly lacks probabilistic outputs or uncertainty quantification (Li et al., 16 Feb 2025). MeteorPred, M3R, and FuXi-Air all note the need for additional multi-source data, broader spatial coverage, or more explicit physical constraints (Tang et al., 9 Aug 2025, Panta et al., 15 Apr 2026, Geng et al., 9 Jun 2025). Weather-R1 and K-MetBench show that reasoning quality cannot be inferred from answer accuracy alone, which makes explanation faithfulness a genuine safety issue rather than a stylistic preference (Wu et al., 20 Jan 2026, Kim et al., 27 Apr 2026). RadarQA likewise indicates that expert-like forecast assessment requires domain-specific descriptive ability, not only numerical verification (He et al., 17 Aug 2025).

The trajectory of the field is nevertheless relatively coherent. Multiple papers explicitly propose adding text, image, and time-series modalities into shared backbones, integrating physics-informed constraints, extending from regional to larger-scale coverage, and evaluating under few-shot, zero-shot, and out-of-distribution regimes (Chen et al., 2024, Li et al., 16 Feb 2025, Tang et al., 9 Aug 2025, Zhang, 28 Mar 2026, Fu et al., 10 Apr 2025). A plausible synthesis is that future MMLMs will combine modality-specific encoders, shared multimodal transformers, adapter or low-rank specialization, expert-verified reasoning supervision, and benchmark suites that test not only forecast skill but also chart literacy, locality, and logical faithfulness. Current work already supplies most of these ingredients, but not yet in a single unified system.