---
title: 'RadarQA: Radar Forecast Quality Analysis'
url: https://www.emergentmind.com/topics/radarqa
type: topic
---

# RadarQA: Radar Forecast Quality Analysis

RadarQA is a multi-modal quality analysis framework for weather radar forecasts that uses Multi-modal Large Language Models to perform both quantitative rating and qualitative assessment of forecast quality. It was introduced to address a limitation of traditional score-based evaluation metrics, which can quantify certain forecast errors but are described as being far from meteorological experts in descriptive capability, interpretability, and understanding of dynamic evolution. The framework integrates key physical attributes with detailed assessment reports, defines a task paradigm covering both single-frame and sequence analysis under rating and assessment scenarios, and is supported by the RQA-70K dataset and a multi-stage training strategy [2508.12291].

## 1. Problem setting and conceptual scope

RadarQA is situated in weather forecast evaluation rather than weather generation. Its central premise is that forecast quality cannot be adequately characterized by scalar metrics alone when meteorological practice also depends on descriptive judgment, temporal reasoning, and physically grounded interpretation. In this setting, RadarQA treats quality analysis as a multi-modal task in which radar imagery or radar sequences are jointly processed with textual evaluation prompts, and the output may be either a discrete rating or a natural-language report [2508.12291].

The framework formalizes four evaluation settings. At the frame level, it supports both frame rating and frame assessment. At the sequence level, it supports sequence rating and sequence assessment. Frame rating assigns discrete ratings for static forecast qualities, whereas sequence rating targets dynamic and cumulative properties across time. Frame assessment and sequence assessment generate textual reports that describe strengths, deficiencies, and temporal evolution in expert-style language. This design places qualitative and quantitative evaluation within a single task family rather than treating them as unrelated post hoc analyses.

A defining feature of RadarQA is its emphasis on physically meaningful attributes. Instead of asking only whether predicted and observed radar fields match numerically, it asks whether the forecast preserves high-value regions, avoids false alarms, maintains sharpness, and captures dynamic consistency. This suggests a shift from purely score-centric validation toward analysis that is closer to operational meteorological reasoning, while remaining structured enough for model training and benchmarking.

## 2. Model structure and task interface

RadarQA is built upon MLLMs capable of processing both visual and textual modalities, and its implementation uses Qwen-2.5-VL-7B as the base model [2508.12291]. The system takes as input a pair of ground truth and predicted radar image frames or sequences, together with textual queries that specify the evaluation task. Depending on the task, the model outputs either discrete quantitative ratings such as “High Value Match: Fair” or detailed qualitative assessment reports in natural language.

The task interface is intentionally uniform across static and temporal analysis. A single predicted frame can be rated across four main attributes—Miss, False Alarm, Sharpness, and High Value Match—plus an Overall rating. A sequence can be rated across three primary dynamic attributes—Dynamic Consistency, Cumulative Precipitation, and High Value Retain—plus Overall. The assessment variants replace categorical labels with free-form diagnostic text.

| Task | Output type | Train / Val |
|---|---|---|
| Frame Rating | Discrete ratings | 20,000 / 860 |
| Frame Assessment | Textual report | 14,500 / 410 |
| Sequence Rating | Discrete ratings | 20,000 / 801 |
| Sequence Assessment | Textual report | 14,500 / 179 |

This task structure makes RadarQA both a benchmark and a modeling framework. It can evaluate point-in-time radar forecast quality, but it can also analyze the evolution of forecast skill over time. In operational terms, the distinction matters because many forecast failures are not strictly local pixel errors; they involve mis-tracked convective motion, loss of organization, or under-retention of intense precipitation cores.

## 3. Attribute system and RQA-70K annotation pipeline

RadarQA uses a physical-attribute-focused library organized into five super-categories: Intensity, Precipitation Conservation, Precipitation Dynamic Distribution, Convective Cycle, and Morphology [2508.12291]. Examples listed for these categories include Miss, False Alarm, High Value Retain or Match, Cumulative Precipitation, Relative Precipitation Error, Morphogenesis, Trajectory, Dynamic Consistency, Genesis or Dissipation of Cells, Sharpness, and Organization Degree.

For rating tasks, seven core attributes are used: Miss, False Alarm, Sharpness, High Value Match, Dynamic Consistency, Cumulative Precipitation, and High Value Retain. Frame-level attributes focus on static spatial qualities, while sequence-level attributes focus on evolutionary and temporal behavior. This decomposition gives RadarQA a domain-specific ontology of forecast quality that is richer than a single scalar score.

The RQA-70K dataset is constructed through a hybrid annotation pipeline that combines human expert labeling with automated heuristics. Perception-based attributes, reported as 17 in total, are annotated by trained human experts using detailed questionnaires. Metric-based attributes, reported as 20 in total, are computed automatically using scripts and expert-defined thresholds or rules. The paper gives explicit examples:

\[
\text{FAR} = \frac{|\mathcal{P}\setminus\mathcal{G}|}{|\mathcal{G}\cap\mathcal{P}| + |\mathcal{P}\setminus\mathcal{G}|}
\]

\[
\text{Miss Rate} = \frac{|\mathcal{G}\setminus\mathcal{P}|}{|\mathcal{G}\cap\mathcal{P}| + |\mathcal{G}\setminus\mathcal{P}|}
\]

\[
\mathcal{E}_{rel} = \left| \frac{N_{gt} - N_{pred}}{N_{gt}} \right|,\quad \text{score} = \max(0, 1 - \mathcal{E}_{rel})
\]

Sharpness is described as being based on Sobel filter difference between prediction and ground truth. Assessment reports are then generated by feeding annotated attributes into GPT-4o, with the stated aim of ensuring fluency and multi-perspective analysis. The raw data source is derived from the SEVIR dataset, and predictions from seven state-of-the-art radar nowcasting models are used to populate the benchmark.

This annotation design is notable because it separates visually judged meteorological phenomena from scriptable error patterns, while still bringing both into a common supervised format. A plausible implication is that RadarQA is designed not only to imitate expert language, but also to anchor that language in a partially formalized attribute space.

## 4. Training strategy and optimization objectives

RadarQA uses a three-phase training pipeline: supervised fine-tuning, reinforcement learning, and post-training [2508.12291]. In the supervised fine-tuning stage, the model is trained on the RQA-70K data using LoRA parameter-efficient fine-tuning, with low-rank matrices trained over frozen model parameters. The reported LoRA rank is 8 for main training.

The second stage is reinforcement learning via GRPO*. The supervised model serves as the initial policy, and two reward functions are used. The format reward enforces a well-structured JSON output, with \( r_{fmat}=1 \) if all keys are correct. The accuracy reward measures alignment between model-predicted intensity labels and annotated ground truth, defined as \( r_{acc} = N_{hit}/N_{all} \). The stated objective is to encourage more accurate and interpretable rating decisions.

The third stage is post-training, described as a final LoRA fine-tuning step on a small, high-quality subset to refine output formats and model alignment. In this stage, the LoRA rank is reduced to 4. The training configuration also includes radar data preprocessing in which data are discretized into 6 precipitation levels and colored per SEVIR conventions, and the reported training hardware is 8 NVIDIA A800 GPUs for 50 hours.

The overall training design indicates that RadarQA is not optimized solely for answer correctness in a narrow classification sense. It is also optimized for response structure, output regularity, and compatibility with downstream evaluation. This suggests that the framework treats expert-style forecast analysis as a constrained generation problem in which formatting, attribute fidelity, and descriptive adequacy are jointly relevant.

## 5. Empirical performance and evaluation

RadarQA is evaluated on rating tasks with accuracy and on assessment tasks with BLEU, ROUGE-L, METEOR, BERTScore, and human GPT-4 grading out of 10 [2508.12291]. The reported quantitative results place the model ahead of the general MLLM baselines included in the study.

For frame rating accuracy, RadarQA reports 61.51% overall, compared with 52.67% for the best API-based model, o1, and 23.76% for the best open-source MLLM, Qwen2.5-VL-72B. For sequence rating accuracy, RadarQA reports 66.17% overall, compared with 45.00% for GPT4o and 36.70% for InternVL2.5-8B. For qualitative assessment, the reported GPT-4 score is 6.87 for frame assessment and 6.58 for sequence assessment, compared with best baselines of 5.89 and 5.63 respectively.

| Setting | RadarQA | Best baseline |
|---|---:|---:|
| Frame Rating Accuracy | 61.51% | 52.67% |
| Sequence Rating Accuracy | 66.17% | 45.00% |
| Frame Assessment GPT-4 Score | 6.87 | 5.89 |
| Sequence Assessment GPT-4 Score | 6.58 | 5.63 |

The paper also reports robust out-of-distribution behavior on new radar tasks such as reflectivity synthesis via models not seen in training. Ablation studies indicate that supervised fine-tuning, reinforcement learning, and post-training each contribute to improvement, and that multi-task joint training on all four tasks outperforms independent single-task variants. A human expert study further reports that meteorologists rated RadarQA’s reports as superior to GPT-4o and highly aligned with ground truth labels, particularly on information density and relevance to expert concerns.

These results are presented as evidence that the model’s gains are not restricted to categorical label prediction. They extend to report quality, domain alignment, and robustness under task transfer.

## 6. Position in the literature, limitations, and later developments

RadarQA is explicitly presented as an alternative to legacy evaluation practices based on metrics such as CSI, SSIM, and RMSE, which it characterizes as insufficient for expert-level descriptive analysis and temporal understanding [2508.12291]. Its main contribution within this literature is to define a holistic evaluation paradigm that combines frame-level and sequence-level analysis, rating and free-text assessment, and physical attributes with language outputs. In that sense, it serves both as a model and as a benchmark infrastructure for weather forecast quality analysis.

The paper also states several limitations. RadarQA is only evaluation, not yet designed for side-by-side model comparison. Fine-grained descriptions can improve further. The possible use of assessment outputs as feedback for improving forecast models remains unexplored. These limitations are important because they delimit the system’s role: it is an evaluator and interpreter of forecast quality, not yet a direct controller for forecast model optimization.

Subsequent work places RadarQA within a broader movement toward unified weather models. “Omni-Weather: Unified Multimodal Foundation Model for Weather Generation and Understanding” describes RadarQA as part of the weather understanding task family and reports that Omni-Weather beats RadarQA on radar image and sequence attribute accuracy and dynamic consistency. That work frames RadarQA as an earlier specialized system and benchmark for understanding, whereas Omni-Weather seeks to unify weather generation and understanding in a single architecture [2512.21643].

In that broader history, RadarQA can be understood as a domain-specialized precursor to unified multimodal weather reasoning systems. Its distinct contribution is not forecast generation, but the formalization of weather-radar quality analysis as a multimodal, expert-aligned task with structured attributes, natural-language assessments, and a large-scale benchmark.

Source: https://www.emergentmind.com/topics/radarqa