CLEARS 2025 Subtask 1 Overview

• CLEARS 2025 Subtask 1 is a benchmark designation applied across multiple domains, with task-specific formulations and metrics for structured evaluation.
• It encompasses areas like tabular QA, plain language rewriting, visual medical QA, hate speech modeling, dialect identification, cultural MCQs, inheritance reasoning, and autonomous driving.
• A common theme is the use of intermediate representations and domain adaptation techniques to improve accuracy, error analysis, and compute efficiency in complex tasks.

“CLEARS 2025 Subtask 1” denotes a recurring first-track designation across several heterogeneous 2025 evaluation settings rather than a single uniform benchmark. In the supplied literature, the label is attached to question answering over tabular data, Spanish plain-language rewriting, gastrointestinal endoscopic visual question answering, Bangla hate-speech modeling, spoken Arabic dialect identification, Arabic and Islamic culture multiple-choice evaluation, Islamic inheritance reasoning, and end-to-end autonomous driving (Lagos et al., 28 May 2025, Site et al., 1 Aug 2025, Moreno et al., 21 Sep 2025, Ayesh et al., 5 Aug 2025, Gaihre et al., 19 Jul 2025, Riad, 9 Jan 2026, Hoque et al., 23 Nov 2025, Talafha et al., 2 Sep 2025, Alwajih et al., 2 Sep 2025, Ahmad et al., 28 Sep 2025, Xing et al., 4 Jun 2026). This suggests that “Subtask 1” functioned as a stable organizational label for the primary track within distinct benchmark families, while the underlying task definition, output space, and scoring rule remained domain-specific.

1. Benchmark landscape and problem scope

Across the cited papers, Subtask 1 is consistently the main entry point of a benchmark family, but the underlying computational problem varies sharply. Some formulations are generative, some discriminative, some executable, and some closed-loop.

Benchmark family	Subtask 1 formulation	Main reported metric
SemEval-2025 Task 8 / DataBench	Question-answering over tabular data	Accuracy
CLEARS Spanish adaptation	Adaptation of text to Plain Language in Spanish	SIM and Fernández-Huerta
ImageCLEFmed MEDVQA-GI 2025	Visual question answering for GI endoscopy	BLEU, ROUGE, METEOR
BLP-2025 Task 1	Bangla hate-speech type / target-group classification	micro-F1
NADI 2025	8-way spoken Arabic dialect identification	Accuracy and $C_{avg}$
PalmX 2025	General Arabic Culture / General Islamic Culture MCQs	Accuracy
QIAS 2025	Islamic Inheritance Reasoning MCQs	Accuracy
NAVSIM v1 setting in CLEAR	End-to-end autonomous driving trajectory planning	PDMS

In the tabular QA papers, Subtask 1 is framed as “Question-Answering over Tabular Data,” where answers must be recovered from a provided table and matched to accepted output types rather than produced as unconstrained prose (Lagos et al., 28 May 2025, Site et al., 1 Aug 2025). In the Spanish adaptation papers, Subtask 1 is the Plain Language track, centered on rewriting municipal or public-administration texts while preserving meaning and improving readability (Moreno et al., 21 Sep 2025, Ayesh et al., 5 Aug 2025). In MEDVQA-GI, Subtask 1 is a multimodal medical VQA problem over gastrointestinal endoscopy images (Gaihre et al., 19 Jul 2025).

The classification-oriented instantiations are equally varied. In BLP-2025 Task 1, the first task concerns Bangla hate-speech analysis, with papers addressing hate-type classification and target-group identification over Bangla social media text and YouTube comments (Riad, 9 Jan 2026, Hoque et al., 23 Nov 2025). In NADI 2025, Subtask 1 is spoken dialect identification over eight country-level Arabic dialect labels (Talafha et al., 2 Sep 2025). In PalmX 2025, Subtask 1 is “General Arabic Culture,” a multiple-choice benchmark for culturally grounded knowledge in Modern Standard Arabic; the paired Islamic-knowledge track is structurally analogous (Alwajih et al., 2 Sep 2025). QIAS 2025 defines Subtask 1 as Islamic inheritance reasoning over scholar-verified MCQs supported by a fatwa corpus (Ahmad et al., 28 Sep 2025). CLEAR extends the designation to end-to-end autonomous driving, where the first task is reframed as multimodal trajectory generation, adaptive compute allocation, and semantic candidate selection on NAVSIM v1 (Xing et al., 4 Jun 2026).

2. Canonical task formulations and data regimes

The tabular QA instantiations are among the most explicitly formalized. MRT characterizes the benchmark as “Question-Answering over Tabular Data” on DataBench, a benchmark of 65 real-world datasets with more than 1,300 manually crafted QA pairs, and reports train/dev/test splits of 49/16/15 tables and 988/320/522 questions. It restricts final answers to Boolean, String, Number, List of Strings, or List of Numbers, and assumes each question must be answered from the provided table (Lagos et al., 28 May 2025). ITUNLP describes the same Subtask I as zero-shot reasoning over a single relational table per question; in train/dev, fields such as answer, type, columns_used, column_types, sample_answer, and dataset are available, whereas the test split contains only question and dataset (Site et al., 1 Aug 2025).

The Spanish Plain Language track is described in two closely related but not identical dataset accounts. One paper reports 2,400 training text pairs and 607 source texts in the official test set, drawn from official municipal communications published in 2022–2023 by municipalities such as Madrid, Alicante, Benidorm, Orihuela, Torrevieja, and Elda (Moreno et al., 21 Sep 2025). Another describes 3,000 Alicante municipal news items paired with both a Plain Language reference and an Easy-to-Read reference, using a 70/30 split into 2,100 training and 900 test items (Ayesh et al., 5 Aug 2025). Both papers agree that Subtask 1 targets Plain Language rather than the stricter Easy-to-Read format.

The MEDVQA-GI paper defines Subtask 1 as gastrointestinal endoscopy VQA over Kvasir-VQA, described as containing about 58,849 image–question–answer triplets associated with 6,500 high-resolution images, 20 unique question templates, and 502 unique answers. Because of computational constraints, the reported system uses only a 1% stratified subset, split first into 90% training and 10% testing, and then repartitioning the test portion into training and validation (Gaihre et al., 19 Jul 2025).

The Bangla hate-speech systems expose a shared low-resource and imbalanced setting. SyntaxMind reports 35,522 instances each for Subtask 1A and Subtask 1B, with severe skew toward the “None” class in both tasks (Riad, 9 Jan 2026). Gradient Masters frames the same benchmark as Bangla hate-speech identification over YouTube comments, with separate labels for hate type in Subtask 1A and target group in Subtask 1B, and emphasizes spelling variation, transliteration, mixed scripts, and slang (Hoque et al., 23 Nov 2025).

The Arabic speech and culture benchmarks are more tightly bounded. NADI 2025 defines Subtask 1 as an 8-way spoken dialect classification task over Algerian, Egyptian, Jordanian, Mauritanian, Moroccan, Palestinian, Emirati, and Yemeni speech, using an Adaptation set and a blind Test set with non-overlapping series (Talafha et al., 2 Sep 2025). PalmX 2025 defines Subtask 1 as four-option MCQ answering for “General Arabic Culture,” with final split sizes of 2,000 training, 500 development, and 2,000 test questions spanning all 22 Arab countries (Alwajih et al., 2 Sep 2025). QIAS 2025 defines Subtask 1 as Islamic Inheritance Reasoning with 9,450 training questions, 1,500 validation questions, 1,000 test questions, and 32,000 fatwas as an external knowledge base (Ahmad et al., 28 Sep 2025).

3. Dominant methodological paradigms

A central methodological divide is between direct answer generation and constrained intermediate structure. The tabular QA systems both reject pure end-to-end text answering. MRT decomposes inference into a Column Descriptor, Explainer, Coder, Runner, Interpreter, and Formatter. The Column Descriptor computes datatype, unique values, missingness, min/max/mean/std when applicable, frequent values, and an LLM-generated column description; the Explainer turns the question and metadata into natural-language instructions; the Coder emits a single Pandas function with signature def parse_dataframe(df: pd.DataFrame) -> str: ...; the Runner executes the code with retries; the Interpreter infers the answer type; and the Formatter applies final conversions (Lagos et al., 28 May 2025). ITUNLP uses a lighter zero-shot code-generation pipeline: schema prompting, one Python code block, controlled execution, and up to two correction iterations with error messages and faulty code fed back to the model (Site et al., 1 Aug 2025).

The Plain Language systems are prompt-centric rather than execution-centric. HULAT-UC3M compares zero-shot prompting and LoRA fine-tuning over Salamandra-7B-Instruct and RigoChat-7B-v2, with a normalization pipeline handling dates, times, money amounts, percentages, years, ranges, and large numbers. Its final selection is RigoChat-7B-v2 with a dedicated Plain-Language-oriented prompt and no fine-tuning (Moreno et al., 21 Sep 2025). CardiffNLP likewise avoids model fine-tuning, instead iterating through zero-shot, one-shot, and few-shot prompts over LLaMA-3.2 and then Gemma-3. Its most distinctive operational constraint is to force the model to emit the simplification as a Python dictionary value, combined with explicit sentence-level simplification instructions (Ayesh et al., 5 Aug 2025).

The multimodal and classification papers favor domain-specialized architectures. MEDVQA-GI fine-tunes microsoft/Florence-2-base-ft, freezing the ViT-L/14 vision tower and using autoregressive answer generation with domain-specific augmentations tailored to endoscopy images (Gaihre et al., 19 Jul 2025). SyntaxMind uses BanglaBERT embeddings followed by parallel Bi-GRU and CNN branches, each with attention, then feature fusion and dense classification (Riad, 9 Jan 2026). Gradient Masters uses transformer fine-tuning with k-fold cross-validation, light rule-based normalization, and FGSM adversarial training in embedding space, with $\epsilon = 0.1$ and $\alpha = 0.5$ (Hoque et al., 23 Nov 2025).

The Arabic speech and knowledge benchmarks show another recurring pattern: domain adaptation over strong pretrained backbones. NADI’s baseline is SpeechBrain VoxLingua107 ECAPA-TDNN, while the winning system, ELYDATA-LIA, uses Whisper Large-v3 with two-stage fine-tuning, partial encoder freezing, and augmentations such as speed perturbation, added noise, frequency dropping, and chunk dropping (Talafha et al., 2 Sep 2025). PalmX participant systems are constrained to decoder-only models of at most 13B parameters and rely mainly on full fine-tuning or LoRA over models such as NileChat-3B, Fanar-1-9B-Instruct, and ALLaM-7B-Instruct (Alwajih et al., 2 Sep 2025). QIAS adopts a hybrid RAG stack—Arabic preprocessing, BM25 retrieval, dense retrieval with Arabic-Triplet-Matryoshka-V2, reranking with cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, and prompting of Fanar or Mistral with the top five passages (Ahmad et al., 28 Sep 2025). CLEAR uses yet another structured pipeline: frozen Drive-JEPA features, a single-step conditional drift decoder in VAE latent space, Qwen 3.5 0.8B hidden states as semantic embeddings, an Adaptive Scheduler over predefined $(\alpha, N)$ schemes, and a Cross-Attention Scorer for trajectory selection (Xing et al., 4 Jun 2026).

4. Evaluation regimes and reported results

The literature reports task-specific metrics rather than a single CLEARS-wide score. Raw values are therefore not directly comparable across tracks, but they do show the operating range of each Subtask 1 formulation.

Area	Metric	Reported result
DataBench table QA	Accuracy	MRT: 70.50%; ITUNLP best test: 84.67 with DeepSeek-V3
Spanish Plain Language	SIM / FH	HULAT-UC3M: SIM 0.75, FH 69.72; CardiffNLP: final averaged cosine similarity 70%, FH 78.81
MEDVQA-GI	BLEU / ROUGE-L / METEOR	Private test: 0.160 / 0.880 / 0.490
Bangla hate speech	micro-F1	SyntaxMind: 0.7345 in 1A and 0.7317 in 1B; Gradient Masters: 72.33 test in 1A and 73.44 test in the 1B table, with 73.28 cited in the abstract
Spoken Arabic dialect ID	Accuracy / $C_{avg}$	ELYDATA-LIA: 79.8 / 17.88
PalmX MCQs	Accuracy	72.15% on General Arabic Culture; 84.22% on General Islamic Culture
Islamic inheritance reasoning	Accuracy	Fanar + Transformer Tafsir: 45.0%
Autonomous driving	PDMS	CLEAR: 93.7

Within DataBench, MRT reports a 70.50% test score for Subtask 1, while ITUNLP reports substantially higher open-source-model results, including 84.67 test accuracy for DeepSeek-V3 and 84.09 for DeepSeek-R1 on Subtask I (Lagos et al., 28 May 2025, Site et al., 1 Aug 2025). In the Spanish adaptation track, HULAT-UC3M ranks first in semantic similarity with SIM = 0.75 but fourth in readability with FH = 69.72, whereas CardiffNLP reports third place in Subtask 1 with a final averaged cosine similarity of 70% and Fernández-Huerta 78.81 (Moreno et al., 21 Sep 2025, Ayesh et al., 5 Aug 2025).

The multimodal medical and speech benchmarks report stronger task-specific ceilings. MEDVQA-GI reaches private-test BLEU 0.160, ROUGE-1 0.880, ROUGE-L 0.880, and METEOR 0.490 (Gaihre et al., 19 Jul 2025). NADI’s top spoken-dialect system reaches 79.8% accuracy with $C_{avg}=17.88$, a large gain over the 61.1% / 34.22 baseline (Talafha et al., 2 Sep 2025). PalmX reports 72.15% on General Arabic Culture and 84.22% on General Islamic Culture, while QIAS reaches only 45.0% on inheritance reasoning, reflecting the much heavier symbolic burden of that benchmark (Alwajih et al., 2 Sep 2025, Ahmad et al., 28 Sep 2025). CLEAR reports PDMS 93.7 on NAVSIM v1, with particularly strong NC 99.1, DAC 98.8, and TTC 97.2 (Xing et al., 4 Jun 2026).

5. Recurrent failure modes and benchmark-specific bottlenecks

The table QA papers provide the most granular error accounting. MRT attributes 37.66% of errors to wrong instructions from the Explainer, 14.29% to wrong cell-value filtering, 14.29% to wrong code or exceptions, 6.49% and 3.90% to formatting issues depending on category, and 23.38% to “others” such as ambiguity, unclear expected answers, or incorrect ground truth. It further states that value matching failures such as “Biden” versus “Joe Biden” alone explain about 15% of the errors (Lagos et al., 28 May 2025). ITUNLP identifies runtime, syntax, and degenerate-loop errors, and reports that the correction loop roughly halves execution errors on average (Site et al., 1 Aug 2025).

The Plain Language papers converge on a different bottleneck: unstable supervision and metric tension. HULAT-UC3M stresses that training adaptations often did not clearly follow ISO 24495-1:2023, that some adapted texts added information while others omitted content, and that improving Fernández-Huerta often reduced semantic similarity (Moreno et al., 21 Sep 2025). CardiffNLP reports prompt failures such as copying the original sentence, using English, producing the wrong format, hallucinating dates and numbers, or copying few-shot examples (Ayesh et al., 5 Aug 2025).

Reasoning-heavy MCQ tasks expose the limits of retrieval and similarity. QIAS identifies three main failure modes in inheritance reasoning: complex fractional reasoning, relevant but incomplete retrieved passages, and logical inference beyond retrieval (Ahmad et al., 28 Sep 2025). PalmX observes that general Arabic culture is harder than Islamic knowledge because it is more heterogeneous, more region-specific, and less canonical, while the utility of data augmentation is domain-dependent (Alwajih et al., 2 Sep 2025).

The classification and speech benchmarks are dominated by imbalance and coverage effects. SyntaxMind notes severe class imbalance, especially for Sexism in Subtask 1A (Riad, 9 Jan 2026). Gradient Masters shows that minority classes such as Sexism, Community, and Society are frequently collapsed into None, and that external data augmentation from public Bangla hate-speech datasets helped development performance but hurt test performance because of domain mismatch (Hoque et al., 23 Nov 2025). NADI explicitly warns that not all Arabic dialects are represented, that country-level labels simplify a continuous dialect space, and that real-world speech conditions such as background noise, disfluencies, accented speech, and code-switching remain underrepresented (Talafha et al., 2 Sep 2025). MEDVQA-GI reports that heavy augmentation, including vertical flip and random rotation, degrades performance relative to standard and fine-tuned augmentation, and that the model is stronger on spatial or binary questions than on abstract or procedural ones (Gaihre et al., 19 Jul 2025). CLEAR identifies a different pair of limitations: the scheduler operates over a discrete grid of $(\alpha, N)$ schemes, and the overall system requires multi-stage training (Xing et al., 4 Jun 2026).

6. Cross-cutting significance and research directions

Several cross-task regularities recur despite the heterogeneity of Subtask 1. First, many successful systems replace unconstrained generation with an explicit intermediate representation: Pandas code in tabular QA, dictionary-valued outputs in simplification, cross-encoder-ranked passages in RAG, adjective bottleneck vectors in interpretable sexism detection, or predefined $(\alpha, N)$ schemes in autonomous driving (Lagos et al., 28 May 2025, Site et al., 1 Aug 2025, Ayesh et al., 5 Aug 2025, Ahmad et al., 28 Sep 2025, Xing et al., 4 Jun 2026). This suggests that Subtask 1 settings repeatedly reward controllability and typed structure over free-form generation.

Second, domain-specific adaptation is consistently privileged over generic model scale. QIAS reports that Fanar is preferable to a generic model for Islamic inheritance reasoning, even though retrieval gains remain limited (Ahmad et al., 28 Sep 2025). PalmX shows NileChat-3B reaching the top cultural score through task-specific fine-tuning, while LoRA emerges as the predominant and most effective strategy among participants (Alwajih et al., 2 Sep 2025). SyntaxMind chooses BanglaBERT because general pretrained models can exhibit language-specific bias (Riad, 9 Jan 2026). HULAT-UC3M favors RigoChat-7B-v2 because it is Spanish-specialized (Moreno et al., 21 Sep 2025). MEDVQA-GI emphasizes augmentations that preserve mucosal texture, bleeding points, lesions, and anatomical orientation rather than generic image perturbation (Gaihre et al., 19 Jul 2025).

Third, compute awareness is not incidental but architectural. MRT switches from per-question sequential execution to batch-by-step execution so that only the model needed for the current stage remains loaded on a single RTX A6000 (48 GB) (Lagos et al., 28 May 2025). HULAT-UC3M uses LoRA on a single NVIDIA L4 GPU with 23 GB VRAM (Moreno et al., 21 Sep 2025). Gradient Masters reports dual NVIDIA T4 constraints and explicitly rejects more expensive adversarial methods such as PGD or AWP (Hoque et al., 23 Nov 2025). CLEAR makes compute allocation itself a learned subproblem through adaptive selection of sample count $N$ and conditioning coefficient $\alpha$ (Xing et al., 4 Jun 2026).

Finally, the future-work statements are notably consistent in one respect: they call for stronger grounding than current automatic metrics or generic prompting can provide. MRT proposes reducing losses introduced between pipeline stages and moving toward a less code-centric, more linguistically robust approach (Lagos et al., 28 May 2025). The Spanish adaptation papers call for richer evaluation beyond cosine similarity and Fernández-Huerta, ideally with stronger user-centered validation and closer alignment with ISO 24495-1:2023 (Moreno et al., 21 Sep 2025, Ayesh et al., 5 Aug 2025). MEDVQA-GI points to visual grounding, uncertainty handling, semantic medical reasoning, and multi-turn conversational VQA (Gaihre et al., 19 Jul 2025). NADI points to broader dialect coverage and more robust dialect-aware speech technology (Talafha et al., 2 Sep 2025). Taken together, these directions indicate that CLEARS 2025 Subtask 1, across its many instantiations, served primarily as a testbed for controlled adaptation under domain-specific constraints rather than as a single monolithic task.