---
title: CLARITY Dataset Overview
url: https://www.emergentmind.com/topics/clarity-dataset
type: topic
---

# CLARITY Dataset Overview

The CLARITY Dataset is a hierarchically annotated resource for the evaluation and modeling of response clarity, ambiguity, and evasion in political question-answer (QA) exchanges. Developed for clarity detection and classification benchmarks, the dataset consists of thousands of QA pairs drawn from U.S. presidential interviews and is the basis of the SemEval-2026 shared task on identifying clarity and ambiguity in political discourse [2409.13879][2601.08176]. It provides fine-grained annotation of evasion strategies and has catalyzed research in prompt-based clarity evaluation for both human and large language model (LLM)-based annotators.

## 1. Dataset Composition and Annotation Protocol

The CLARITY Dataset comprises QA pairs extracted from televised interviews, press conferences, and debate transcripts with recent U.S. presidents (Bush, Obama, Trump, Biden; coverage: 2006–2023). After de-duplication, the main release contains 2,047 evaluated QA instances and maintains the following splits:

| Split      | Count (initial/cleaned) |
|------------|------------------------|
| Train      | 3,448 / —              |
| Test       | 308 / —                |
| Unique QA  | — / 2,061              |
| Scored     | — / 2,047              |

Each example is single-barreled, with “multi-question” exchanges programmatically decomposed into singular QA pairs using gpt-3.5-turbo, supplemented by manual validation and counterfactuals. Three trained annotators (plus one expert resolver) annotate every instance for both clarity and evasion. Gold labels are assigned by majority vote. Reported inter-annotator agreement is substantial: κ ≈ 0.70–0.80 for clarity, κ ≈ 0.60 for evasion [2601.08176].

## 2. Taxonomy and Label Distribution

CLARITY employs a hierarchical taxonomy:

**Top-Level Clarity Labels**:
1. Clear Reply: Single, unequivocal answer to question.
2. Ambivalent Reply: Reply accommodates multiple plausible interpretations.
3. Clear Non-Reply: Explicit rejection, ignorance, or request for clarification.

**Fine-Grained Evasion/Evasion Techniques** (nine categories):
- Explicit
- Implicit
- Dodging
- Generalization
- Deflection
- Partial/Half-Answer
- Declining to Answer
- Claims Ignorance
- Clarification Request

The label distribution (n=2,047):

| Clarity           | Count | Percentage |
|-------------------|-------|------------|
| Clear Reply       | 1,000 | 48.8%      |
| Ambivalent Reply  |   800 | 39.1%      |
| Clear Non-Reply   |   247 | 12.1%      |

| Evasion                 | Count | %      |
|-------------------------|-------|--------|
| Explicit                | 500   | 24.4%  |
| Implicit                | 200   | 9.8%   |
| Dodging                 | 150   | 7.3%   |
| Generalization          | 220   | 10.7%  |
| Deflection              | 180   | 8.8%   |
| Partial / Half-Answer   | 160   | 7.8%   |
| Declining to Answer     | 250   | 12.2%  |
| Claims Ignorance        | 200   | 9.8%   |
| Clarification           | 187   | 9.1%   |

CLARITY also annotates topic (14 predefined policy domains plus “Other”) for each QA instance [2601.08176].

## 3. Annotation Guidelines and Quality Metrics

Annotation is governed by a tutorial with model examples for all leaf labels. Multi-expert adjudication resolves disagreements. The QA decomposition protocol includes deliberate counterfactual (misleading) decompositions to ensure focus on context. Agreement is quantified by Fleiss’ κ:

$$
\kappa = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}
$$

with $\bar{P}$ the mean observed agreement and $\bar{P}_e$ the expected chance agreement [2409.13879].

Performance on human and LLM classification is measured by accuracy, per-class precision/recall/F$_1$, macro F$_1$, weighted F$_1$ (weighted by class occurrence), and Hierarchical Exact Match (HEM), i.e., joint correctness across both hierarchy levels [2601.08176].

## 4. Model Baselines and Evaluation

CLARITY provides strong benchmarks for both encoder-based and LLM architectures. Baselines span:
- Encoder models: DeBERTa-base/large, RoBERTa-base/large, XLNet-base/large, fine-tuned for clarity and evasion.
- LLMs: Llama 2 (7B–70B), Falcon (7B/40B), ChatGPT (gpt-3.5-turbo).
- Adaptation: zero-shot, few-shot (one-shot per leaf label), chain-of-thought (CoT) prompting, LoRA instruction tuning.

Key results for 3-way clarity recognition (test set, macro F$_1$):

| Model/Method                | Accuracy | F$_1$ (macro) |
|-----------------------------|----------|--------------|
| ZS ChatGPT (direct)         | 0.649    | 0.413        |
| CoT ChatGPT (evasion)       | 0.688    | 0.510        |
| LoRA Llama-2-70B (direct)   | 0.759    | 0.680        |
| LoRA Llama-2-70B (evasion)  | 0.713    | 0.710        |
| XLNet-base (direct)         | 0.694    | 0.518        |
| RoBERTa-base (direct)       | 0.640    | 0.530        |

Prompt design substantially boosts clarity accuracy (e.g., GPT-5.2 with CoT + 3-shot: 0.59 → 0.60 → 0.64). For fine-grained evasion, the highest accuracy is 0.34 (GPT-5.2 CoT). The explicit class is most reliably detected (precision 0.82, recall 0.42); other categories, such as “Partial/Half-Answer” and “Dodging,” remain error-prone.

## 5. Prompt Engineering and Automatic Model Evaluation

Research demonstrates that chain-of-thought and in-context demonstration prompts systematically improve LLM clarity classification. Explicit reasoning steps facilitate fine-grained distinctions—especially between Ambivalent and Clear Non-Reply categories. However, prompt engineering alone is insufficient to resolve confusions among nuanced evasion strategies (e.g., Implicit vs Dodging) [2601.08176].

Topic identification is also evaluated. Accuracy for topic assignment increases from 60% (simple prompt) to 74% (CoT), but overlapping categories (e.g., Economy vs. Healthcare) remain problematic.

## 6. Limitations, Observed Patterns, and Future Work

CLARITY is restricted to English and U.S. political discourse; non-verbal and paralinguistic signals, which often delineate evasion, are not captured. The reliance on LLM-driven preprocessing for QA decomposition can introduce errors. Annotator expertise is intentionally generalized, but future releases may explore multi-perspective or expert-only annotation protocols.

Patterns noted include higher explicit-reply rates for single-part QAs, and presidency-based variability (e.g., Trump providing the most explicit replies). Answer grounding with multi-part QAs disproportionately challenges LLMs—in some cases lowering F$_1$ by up to 0.16—whereas human annotator agreement is less affected (Δκ ≈ 0.03). Use of person names increases model error, suggesting significant reliance on pre-encoded knowledge in LLMs [2409.13879].

Directions for further study include cross-lingual extension, multimodal (audio/video) integration for richer evasion detection, and adversarial settings where models themselves generate ambiguous answers.

## 7. Applications and Usage

CLARITY enables fine-grained, large-scale analysis of evasive and ambiguous tactics in political interviews, robust AI fact-checking and monitoring pipelines, and research into QA model robustness. The taxonomy and self-contained evaluation framework provide a rigorous empirical foundation for measuring “clarity” and “evasion” across models, datasets, and languages, with all code, prompts, and data splits made publicly available [2409.13879][2601.08176].

Source: https://www.emergentmind.com/topics/clarity-dataset