---
title: 'RQA-70K: Multi-modal Radar Forecast Analysis'
url: https://www.emergentmind.com/topics/rqa-70k
type: topic
---

# RQA-70K: Multi-modal Radar Forecast Analysis

RQA-70K is a large, domain-specific, multi-modal dataset introduced for training and benchmarking Multi-modal Large Language Models (MLLMs) on expert-aligned quality analysis of weather radar forecasts [2508.12291]. It was designed to move forecast verification beyond score-only evaluation by coupling visual radar inputs with structured ratings and detailed assessment reports that target physically meaningful attributes of convective systems, including morphology, dynamic evolution, cumulative precipitation behavior, and high-value retention. The dataset supports both single-frame and sequence-level reasoning, and both rating and assessment outputs, thereby aligning model supervision with meteorological expert workflows.

## 1. Definition, purpose, and scope

RQA-70K was created to support quality analysis that goes beyond pixel-wise scores and to address three gaps in forecast verification: descriptive capability, interpretability, and temporal understanding [2508.12291]. In this setting, descriptive capability refers to characterizations such as “scattered and block-like”; interpretability refers to explaining why a forecast is poor or good; and temporal understanding refers to phenomena such as genesis, dissipation, and movement. The dataset therefore targets a regime in which an MLLM must reason jointly over predicted radar products, corresponding ground truth, question prompts, and attribute-conditioned answer targets.

The dataset is built around visual weather radar inputs derived from the Vertically Integrated Liquid (VIL) modality from SEVIR. VIL values are discretized into six precipitation levels, following SEVIR practice, and colorized to RGB using the SEVIR colormap for visualization. The data focus on storm events over the CONUS region, specifically flash flood, flood, funnel cloud, hail, heavy rain, thunderstorm wind, and tornado events. In the RawRQA-20K base set, the reported event counts are flash flood 218, flood 121, funnel cloud 58, hail 556, heavy rain 55, thunderstorm wind 1030, and tornado 121.

RQA-70K also incorporates forecast diversity by drawing predictions from seven representative nowcasting models: EarthFormer, PredRNN, Cascast, DGMR, Diffcast, SimVP, and NowcastNet. The paper states that these models span GANs, RNNs, diffusion, and transformer-based approaches, with the explicit goal of covering diverse error modes such as artifacts, blurring, and over-smoothing.

## 2. Task structure and supervision formats

RQA-70K covers four tasks that mirror how experts evaluate single frames and sequences, first by rating general attributes and then by producing assessments [2508.12291].

| Task | Input | Output |
|---|---|---|
| Task-1 Frame Rating | One predicted radar image, its corresponding ground-truth image, and one rating question from a 10-question pool | Structured JSON with five attributes |
| Task-2 Frame Assessment | The same image pair and one assessment question from a 10-question pool | Detailed textual report |
| Task-3 Sequence Rating | Predicted sequence, ground-truth sequence, and a rating question | Structured JSON with four attributes |
| Task-4 Sequence Assessment | Predicted and observed sequences plus an assessment question | Multi-paragraph analysis |

For Frame Rating, the JSON output contains discrete levels for Miss, False Alarm, Sharpness, High Value Match, and Overall. For Sequence Rating, the JSON output contains discrete levels for Dynamic Consistency, Cumulative Precipitation, High Value Retain, and Overall. In both rating settings, the rubric uses four performance levels: “Great,” “Good,” “Fair,” and “Poor.”

The assessment tasks are more linguistically demanding. Frame Assessment requires a report that first describes the observation and then assesses misses, false alarms, sharpness, and high-value match, including directional and intensity detail, before ending with a summary statement. Sequence Assessment requires a multi-paragraph analysis that describes observation content and evaluates movement, genesis and dissipation, scale changes, cumulative precipitation, and high-value retention, again with a final summary.

A notable structural design is that each sequence sample can yield one randomly selected frame for the frame tasks. This couples frame-level and sequence-level supervision while preserving coverage of both static and dynamic forecast properties.

## 3. Composition, provenance, and data organization

The dataset contains approximately 69,000 samples, which is why it is referred to as “RQA-70K” [2508.12291]. Of these, 40,000 are brief templated samples for the rating tasks and 29,000 are detailed assessment samples for the assessment tasks.

| Subset | Train | Validation |
|---|---:|---:|
| Task-1 Frame Rating | 20,000 | 860 |
| Task-2 Frame Assessment | 14,500 | 410 |
| Task-3 Sequence Rating | 20,000 | 801 |
| Task-4 Sequence Assessment | 14,500 | 179 |

The validation counts are specified, but the test split is not specified in the paper. Typical sequence structure consists of inputs of 10 frames and targets of 12 frames, while predictions are 12-frame sequences from the seven nowcasting models. For automated metric computation, a precipitation “presence” threshold of 16 is used for binary masks, and high-value pixels are defined by intensity greater than 219.

The basic sample structure differs by task type. Rating samples pair images or sequences with a question string and a ground-truth JSON keyed by the relevant general attributes plus Overall. Assessment samples pair images or sequences with a question string and a ground-truth assessment text. The paper also provides illustrative record layouts in which rating answers are dictionary-valued and assessment answers are free-form reports.

Preprocessing and quality control are explicitly described. VIL is discretized and colorized for visualization; for frame tasks, a single frame is randomly selected from each predicted sequence; and expert spot-checking is routinely performed. For human-labeled batches, domain experts randomly review 35% of samples and accept or reject entire batches based on quality.

## 4. Attribute schema and annotation methodology

RQA-70K is constructed through an Attribute-Informed Generation workflow in which labels for key attributes are first collected and text is then generated conditioned on those attributes [2508.12291]. The underlying scientific attribute library comprises five super-categories, 10 sub-categories, and 35 attributes total, split into 15 frame-level and 22 sequence-level attributes, plus “Overall” at frame and sequence levels, for 37 attributes used.

Seven “general attributes” are used directly in the rating tasks:

- **Miss**: proportion of observed convective regions not captured by prediction.
- **False Alarm**: proportion of predicted convective regions not present in ground truth.
- **Sharpness**: clarity of predicted convective boundaries.
- **High Value Match**: alignment of high-intensity cores between prediction and observation.
- **Dynamic Consistency**: correct capture of evolution, including movement speed, genesis, and dissipation.
- **Cumulative Precipitation**: match of temporally integrated precipitation.
- **High Value Retain**: preservation of high-intensity regions over time.

The annotation pipeline is hybrid. Seventeen perception-based attributes are human annotated. Experts designed multiple-choice questionnaires with clear guidelines, golden examples, and training materials; annotators then underwent training and a pilot phase, cross-validation among annotators, and batch-level expert validation. The perception-based attributes include shape and scale changes, movement direction, speed differences, rotation center, genesis and dissipation differences, organization degree, artifacts direction, and distribution. Inter-annotator agreement values are not specified.

Twenty attributes are annotated automatically using domain-expert-defined thresholds. The automated layer formalizes several general attributes through explicit metrics. Let $G$ and $P$ denote the sets of pixels with precipitation, thresholded at 16, in ground truth and prediction respectively. Then $H = |G \cap P|$ and $F = |P \setminus G|$, so the false alarm rate is

$$
\text{false alarm rate} = \frac{F}{H + F}.
$$

The thresholds $[0.1, 0.2, 0.3]$ map this quantity to the four performance levels. Miss performance is defined using $M = |G \setminus P|$ and

$$
\text{miss rate} = \frac{M}{H + M},
$$

with thresholds $[0.1, 0.2, 0.4]$.

Sharpness performance is Sobel-based: the method compares the mean Sobel magnitudes of ground truth and prediction, clips negatives to zero, and maps the resulting sharpness score to four levels using thresholds $[0.5, 0.7, 0.9]$. High-value mismatch performance is based on high-value pixels with intensity greater than 219. If $N_{pred}$ and $N_{gt}$ denote the corresponding counts, then the relative error is

$$
\mathcal{E}_{rel} = \left| \frac{N_{gt} - N_{pred}}{N_{gt}} \right|,
$$

and the score is

$$
\text{high value mismatch score} = \min\left(1, \max(0, 1 - \mathcal{E}_{rel})\right).
$$

The thresholds $[0.3, 0.6, 0.8]$ map this score to four levels. Sequence-level High Value Retain is the average of frame-level high-value mismatch scores, using the same thresholds. Cumulative precipitation performance uses the same relative-error and scoring form as above, but with thresholds $[0.93, 0.97, 0.99]$.

Assessment generation is also structured spatially and semantically. Each image is divided into a $3 \times 3$ grid defining nine directional sectors. Descriptions combine direction and issue type, and rainfall levels are weighted by $[1, 1.5, 2.5, 5, 10, 20]$ from “light” to “extreme” for selecting salient levels in miss and false-alarm analysis.

## 5. Text generation, training usage, and benchmark role

The textual assessment targets in RQA-70K are produced through Attribute-Informed Generation using GPT-4o, with explicit instructions and visual context intended to reduce inconsistencies [2508.12291]. This means that the assessment side of the dataset is not purely manual prose; rather, it is generated from the annotated attribute layer. The paper explicitly documents known biases from this choice: style uniformity, occasional misalignment with visuals, redundancy, and omission of less prominent attributes.

RQA-70K is used as the training and evaluation substrate for RadarQA, an MLLM built on Qwen-2.5-VL that takes multi-image inputs together with question prompts. Instruction tuning is performed separately for rating and assessment tasks. The paper describes a three-stage training strategy: supervised fine-tuning with LoRA on the full RQA-70K; reinforcement learning with GRPO on rating tasks; and post-training with low-rank LoRA updates on a small subset to refine outputs and standardize formats.

The GRPO stage uses two rewards on rating tasks. The format reward $r_{fmat}$ is 1 if the response is a valid JSON with all required keys and 0 otherwise. The accuracy reward is

$$
r_{acc} := \frac{N_{hit}}{N_{all}}
$$

if $r_{fmat} = 1$, and 0 otherwise. The paper states that this curriculum improves both numeric ratings and descriptive assessments, and that joint training on all four tasks outperforms single-task training on each task.

Evaluation on RQA-70K is split by task type. Rating tasks use accuracy on each general attribute, computed from structured JSON outputs with predefined keys. Assessment tasks use BLEU, ROUGE_L, METEOR, BERTScore, and a GPT-4 score from 0 to 10 for relevance, accuracy, and detail.

## 6. Benchmark results, limitations, and interpretation

On the RQA-70K validation settings, RadarQA outperforms existing general MLLMs across all evaluation settings reported in the paper [2508.12291].

| Evaluation setting | RadarQA result |
|---|---|
| Frame Rating | Overall 61.51%; False Alarm 65.35%; Miss 67.67%; High Value Match 69.19%; Sharpness 78.60% |
| Frame Assessment | BLEU 0.213; BERTScore 0.809; ROUGE_L 0.512; METEOR 0.420; GPT-4 score 6.87 |
| Sequence Rating | Overall 66.17%; Dynamic Consistency 53.31%; Cumulative Precipitation 48.94%; High Value Retain 80.52% |
| Sequence Assessment | BLEU 0.212; BERTScore 0.815; ROUGE_L 0.436; METEOR 0.461; GPT-4 score 6.58 |

The paper also reports strong ablation gains. Multi-stage training improves average accuracy from 27.79 and 16.20 to 68.46 and 62.24 for frame and sequence rating, and improves GPT-4 scores from 3.81 and 3.92 to 6.87 and 6.58 for frame and sequence assessment. For sequence rating, the paper reports that the open-source baseline InternVL2.5-8B attains overall 36.70%, while API baselines are in the 20–30% range. For frame rating, it notes that general-purpose MLLMs struggle especially on High Value Match, with open-source baselines below 20%.

The dataset has several explicit limitations. It is focused on VIL from SEVIR and convective storm events over CONUS, and the paper states that spatial resolution and cadence are not specified; a plausible implication is that direct transfer to other sensors, such as reflectivity in dBZ, or to regions outside CONUS would require adaptation. The human-annotation layer does not report inter-annotator agreement. The assessment text layer inherits GPT-4o-induced biases. The forecast inputs themselves may contain model-specific artifacts, temporal blurring, or over-smoothing, and the paper cautions that generated analyses should be treated as guidance rather than sole verification.

Responsible-use recommendations are correspondingly explicit. Textual assessments should be cross-checked with standard meteorological metrics and domain experts; users should avoid over-reliance on a single modality or automated thresholds; local climatology and operational context should be considered; and downstream decisions that depend on generated analyses should be documented with appropriate human oversight. Code and dataset access are provided through the RadarQA repository, while licensing and data usage constraints are not specified in the paper.

In this formulation, RQA-70K functions as both a supervision corpus and a benchmark for expert-like, multi-modal quality analysis of weather radar forecasts. Its central contribution is not only scale, but the coupling of computable heuristics, perception-based meteorological attributes, and assessment-style textual targets within a single task framework that spans both static and dynamic forecast evaluation.

Source: https://www.emergentmind.com/topics/rqa-70k