Aha! Datasets: Diverse Data & Evaluation Techniques

• Aha! Datasets are a collection of heterogeneous, domain-specific data artifacts that share a common label but differ in structure, purpose, and application.
• They employ structured perturbations and protocol-level benchmarks—for example, in audio hallucination, robotic failure, and causal reasoning—to ensure alignment and diagnostic consistency.
• These datasets impact multiple fields, enhancing tasks from audio-language model performance and robotic manipulation to medical imaging and online highlight detection.

“Aha! Datasets” does not denote a single canonical corpus. In the literature, the label appears across several unrelated research programs and covers standalone datasets, diagnostic benchmarks, procedural generation pipelines, protocol-level evaluation suites, and derived labeling schemes. The name is used for audio-grounding hallucination alignment resources in large audio-LLMs, robotic manipulation failure trajectories, chain-of-thought causal analyses of “aha moments,” reasoning-first SVG generation corpora, AI-harm anticipation vignettes, online highlight detection data, an Omniglot-derived episodic-learning benchmark, and cardiac imaging outputs mapped to the AHA 17-segment clinical standard (Chen et al., 30 Dec 2025, Duan et al., 2024, Zhao et al., 28 Oct 2025, Xing et al., 30 May 2025, Buçinca et al., 2023, Chang et al., 19 Sep 2025, Kowadlo et al., 2019, Montalvo-García et al., 30 Jun 2026).

1. Scope, naming, and dataset forms

A common misconception is that AHA or Aha identifies a single dataset lineage. The published record indicates otherwise. In one line of work, AHA means Audio Hallucination Alignment and refers to a unified audio–question pool split into an alignment dataset and a diagnostic benchmark for large audio-LLMs. In another, AHA names a vision-LLM and its associated dataset for robotic failure reasoning. Elsewhere, AHA! means Anticipating Harms of AI; AHA denotes an Artificial Hippocampal Algorithm evaluated on an Omniglot extension; and “AHA” in cardiac imaging refers to automatic mapping to the AHA 17-segment clinical standard rather than a naming convention for the dataset itself (Chen et al., 30 Dec 2025, Duan et al., 2024, Buçinca et al., 2023, Kowadlo et al., 2019, Montalvo-García et al., 30 Jun 2026).

The data artifacts also differ structurally. Some are explicitly released training resources, such as the audio alignment dataset and HIHD for online highlight detection. Some are evaluation-only sets, such as ManiSkill-Fail and RoboFail in the robotics paper. Some are protocol-level constructions over pre-existing corpora, such as the Omniglot Extended Benchmark. Others are derived labels layered onto existing data, such as per-frame AHA segment assignments for CAMUS echocardiography. This suggests that “Aha! Datasets” is best understood as a family of domain-specific dataset designs linked by a shared label rather than by a shared ontology.

2. Audio-grounding hallucination datasets

Under the AHA framework for large audio-LLMs, the core resource is a shared audio–question pool $D = \{(a_i, q_i)\}$, built from temporally structured audio clips and hallucination-sensitive question templates. Two views are derived from this pool: $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$ for post-training alignment, and $D^{eval} = \{(a_i, q_i, y_i^*, \tau(q_i))\}$ for diagnostic evaluation. Both target a four-category hallucination taxonomy: Event Omission, False Event Identity, Temporal Relation Error, and Quantitative Temporal Error (Chen et al., 30 Dec 2025).

The source corpus is AudioTime, reported as 20,000 training and 2,000 testing samples spanning duration, frequency, ordering, and timestamp. AHA selects all samples with multiple audio events, producing 11,507 training and 1,251 testing samples in the shared pool. Question generation is programmatic, using predefined templates for explicit ordering, temporal logic and counterfactual reasoning, counting, and duration. GPT-4o is used to instantiate questions and to produce candidate rejected responses containing targeted hallucination types. For alignment, 10 human volunteers select the single most representative hard negative from LLM-generated candidates, prioritizing linguistically plausible but acoustically incorrect responses. For evaluation, 10 human volunteers verify concise ground-truth answers and annotate hallucination type; ambiguous or misaligned instances are discarded (Chen et al., 30 Dec 2025).

The alignment dataset is designed for Direct Preference Optimization on Qwen2.5-Omni-7B with a frozen reference model and LoRA adapters. Its function is to force preference for acoustically grounded responses over plausible fabrications. AHA-Eval, by contrast, is a QA benchmark with concise, human-verified answers and category-wise hallucination rates. Reported results state that Qwen-Audio-AHA achieves a 13.7% improvement on AHA-Eval relative to Qwen2.5-Omni, with category-wise error-rate reductions from 70.6% to 53.8% for Event Omission, 70.6% to 64.1% for False Event Identity, 30.5% to 15.9% for Temporal Relation Error, and 69.6% to 52.6% for Quantitative Temporal Error. The same paper reports gains on public benchmarks: MMAU-test-mini 74.6% to 76.4%, MMAU-test 71.2% to 72.5%, MMAU-Pro 54.2% to 54.8%, and MMAR 58.1% to 59.7% (Chen et al., 30 Dec 2025).

Conceptually, these resources are unusual because training and evaluation are built from the same audio–question pool. The paper explicitly presents this as a “unified two-view construction,” intended to keep alignment signals and diagnostic criteria consistent. A plausible implication is that the dataset design tries to minimize the usual mismatch between preference data and benchmark definition.

3. Failure-centric datasets for robotic manipulation

The robotic AHA dataset is a procedurally generated collection of manipulation failure trajectories built to instruction-tune a vision-LLM for sub-task failure detection and free-form reasoning. Its purpose is to let a model judge whether a sub-task has succeeded with a binary “yes” or “no” and, if failed, produce a natural-language explanation of the cause. The generation framework, FailGen, perturbs successful RLBench demonstrations by editing keyframes, objects, and action sequences in a controlled way (Duan et al., 2024).

The paper defines seven failure modes: Incomplete Grasp (No_Grasp), Inadequate Grip Retention (Slip), Misaligned Keyframe (Translation), Incorrect Rotation (Rotation), Missing Rotation (No_Rotation), Wrong Action Sequence (Wrong_action), and Wrong Target Object (Wrong_object). For each RLBench task, FailGen generates YAML configuration files specifying the failure mode, the keyframe or keyframes to perturb, and numeric parameters controlling the perturbation. The released failure pairs contain RGB image grids composed across viewpoints and temporal keyframes, natural-language prompts, ground-truth answers, and YAML configuration files. The released pairs do not include depth images, segmentation masks, robot proprioception, or low-level action/state logs (Duan et al., 2024).

In scale and split design, the AHA dataset contains 49,000+ failure image–query pairs from 79 RLBench tasks for training and 11,000 image–question pairs from 10 unseen RLBench tasks for testing. Two external evaluation sets are central to the paper’s generalization claims: ManiSkill-Fail, with 130 image–question pairs across 4 ManiSkill tasks, and RoboFail, a real-world UR5 benchmark across 7 tasks. The AHA model is also co-finetuned on 665,000 VQA conversation pairs and 100,000 LVIS-derived detection entries to preserve general visual-linguistic competence (Duan et al., 2024).

Evaluation uses ROUGE-L, cosine similarity, LLM Fuzzy Match, and binary success. On the 11K AHA test set, AHA-13B reports ROUGE-L 0.446, cosine 0.583, binary 0.702, and fuzzy 0.768. On ManiSkill-Fail, AHA-13B reports ROUGE-L 0.600, cosine 0.681, binary 1.000, and fuzzy 0.633. On RoboFail, it reports ROUGE-L 0.280, cosine 0.471, binary 0.643, and fuzzy 0.465. The abstract further states that AHA surpasses GPT-4o in-context learning by 10.3% and exceeds the average performance of six compared models by 35.3% across multiple metrics and datasets. The same paper integrates AHA into reinforcement learning reward design, task and motion planning, and zero-shot trajectory generation, reporting an average task success rate improvement of 21.4% across all three tasks compared to GPT-4 models (Duan et al., 2024).

This dataset family is notable for making failure, rather than success, the primary supervision target. The paper frames this as a route to robustness across robots, viewpoints, simulators, and tasks. It also exemplifies a recurrent AHA pattern: a tightly specified failure taxonomy, procedural perturbations, and natural-language rationales linked to those perturbations.

4. Reasoning-step datasets: causal analysis and Drawing-with-Thought

One AHA-themed line of work studies whether self-verification steps in chain-of-thought are genuine or merely decorative. The paper introduces the True Thinking Score, defined for a step $s$ as

$$\mathrm{TTS}(s)=\tfrac{1}{2}\left(\lvert S_1(1)-S_0(1)\rvert+\lvert S_1(0)-S_0(0)\rvert\right),$$

where the two terms combine necessity under intact context and sufficiency under perturbed context. The empirical setting uses AMC, AIME, and MATH with DeepSeek-R1-Distill-Qwen-7B, its 1.5B variant, DeepSeek-R1-Distill-Llama-3.1-8B, and OpenReasoning-Nemotron-1.5B. The central finding is that chain-of-thought often interleaves true-thinking and decorative-thinking steps; on AIME with Qwen-2.5-7B, mean TTS is approximately 0.03, 6.4% of steps have TTS greater than 0.3, and only 2.3% have TTS at least 0.7. The same paper reports that about 12% of self-verification steps in Qwen-2.5-7B and about 21% in Nemotron-1.5B have TTS below 0.005, indicating near-zero causal influence even when the text appears to “double-check” the solution (Zhao et al., 28 Oct 2025).

The dataset guidance in that paper is explicit. It proposes step-level labels of genuine for TTS at least 0.7, decorative for TTS at most 0.03, and borderline for intermediate values, plus regex or heuristic detection of self-verification segments containing markers such as “wait,” “let me recompute,” “re-evaluate,” “double-check,” or “make sure.” It further recommends storing the original and perturbed contexts, per-step probabilities $S_1(1), S_0(1), S_1(0), S_0(0)$, ATE terms, TTS labels, and optional steering metadata based on a latent “TrueThinking direction” (Zhao et al., 28 Oct 2025).

A different reasoning-oriented resource is SVGX-DwT-10k, introduced for SVG generation under the “Drawing-with-Thought” paradigm. Each of its 10,000 entries is a triplet $(T, C, O)$ consisting of a natural-language prompt, a DwT rationale, and SVG code. Every rationale contains six explicit stages: Concept Sketching, Canvas Planning, Shape Decomposition, Coordinate Calculation, Styling and Coloring, and Final Assembly. The corpus spans four domains—Logo & Emoji, Iconography, UI & Layout, and Diagrams—with icon-style graphics dominating more than 80%. Seventy percent of DwT sequences exceed 1,000 tokens and 10% exceed 3,000 tokens. The paper uses the full dataset for supervised fine-tuning, samples 2,000 prompts for reinforcement-learning prompts, and defines a held-out 1,000-prompt evaluation set (Xing et al., 30 May 2025).

SVGX-DwT-10k is generated with Gemini 2.5 Pro and then manually reviewed and revised by domain experts. Validation includes CairoSVG renderability, semantic faithfulness to the prompt, design consistency, visual coherence, reasoning logic, and internal consistency between the DwT steps and the final SVG. The paper’s full Reason-SVG system reports Val% 99.8, FID 18.6, CLIPScore 0.345, Aesthetic 5.9, HPSv2 21.80, and DwT-Cover% 100, with human evaluation scores of SemAcc 4.53 ± 0.35, VisApp 4.42 ± 0.39, and DwT-Qual 4.61 ± 0.31 (Xing et al., 30 May 2025).

Taken together, these two resources suggest two sharply different dataset roles for explicit reasoning traces. In the TTS work, reasoning text is an object of causal audit. In SVGX-DwT-10k, reasoning text is a supervised intermediate representation meant to improve generation. Both make “aha moments” operational, but one treats them as potentially illusory and the other as a target behavior to be elicited.

5. Scenario-based harms and online highlight detection

AHA! for AI impact assessment produces three linked artifacts for each deployment scenario: ethical matrices and vignettes, harm descriptions, and harm annotations. The scenarios are hiring, loan application, content moderation, communication compliance, and disease diagnosis. Across these five binary-classification settings, the paper reports 4,113 raw harm descriptions generated by crowd workers and GPT-3, with 7% flagged as “not meaningful.” The coding taxonomy contains more than 50 unique harm subcategories grouped into eight high-level categories: Quality of service, Representational, Well-being, Legal and reputational, Allocational, Loss of rights or agency, Social and societal, and Other/unspecified (Buçinca et al., 2023).

The data model is matrix-based. Scenario-specific stakeholder lists are crossed with problematic AI behavior dimensions: false positive versus false negative, one-time versus accumulated, egregious versus unspecified severity, and harm specified versus unspecified. Each sampled vignette is completed by three crowd workers and three GPT-3 generations. The crowd study uses Clickworker with English-speaking judges in North America, manual review for gibberish or irrelevance, a speed check, and an attention check; approximately 40% of task responses were flagged and re-judged. GPT-3 generation uses davinci with temperature 0.95, few-shot harms, and three completions per vignette (Buçinca et al., 2023).

The paper emphasizes source complementarity. Crowds-only and GPT-3-only produce comparable unique subcategory counts, but the combined corpus yields significantly more unique subcategories than either source alone across all scenarios. Semi-structured interviews with responsible AI professionals ($N=9$) found that the systematic approach surfaced stakeholders and harms they believed they would not have thought of otherwise, while also shifting some effort from ideation to review (Buçinca et al., 2023).

A different Aha line addresses streaming video. HIHD, the Human Intuition Highlight Dataset, is introduced for task-conditioned Online Highlight Detection and is constructed from Mr.HiSum benchmark entries by re-retrieving original YouTube videos, discarding those with fewer than 70,000 original views, sampling frames at 1 fps, and generating a synthetic natural-language task objective $Q$ from the video title via query templates. The paper reports 22,463 videos in total. Frame-level relevance labels $r_t$ are obtained by normalizing YouTube “most replayed” engagement counts to $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$0. The training pipeline also adds “quality dropout” segments covering 5–20% of each video’s duration, with degradations such as resolution down/up-sampling plus blur, block noise, color banding, and blackout (Chang et al., 19 Sep 2025).

HIHD is paired with auxiliary supervision from Shot2Story and COIN for informativeness and an auxiliary language-modeling head, and it is evaluated against TVSum, Mr.HiSum, SCOUT, Charades-STA, and YouCook2. On TVSum, Aha reports 91.6 top-5 mAP, $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$1, and $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$2 in zero-shot mode, and 93.0 top-5 mAP after domain-adapted grid search on scoring weights. On Mr.HiSum, it reports 64.19 mAP@50 and 32.66 mAP@15. The paper also reports that removing the task prompt causes TVSum mAP to drop from 93.0 to 81.2, underscoring the role of persistent natural-language conditioning (Chang et al., 19 Sep 2025).

These two dataset families are methodologically different, but both externalize task framing. AHA! uses second-person stakeholder vignettes to enumerate possible downstream harms; HIHD uses a persistent natural-language objective to score each frame in a causal stream. In both cases, the dataset is not just a collection of inputs and labels but a structured representation of “what matters” for a particular decision problem.

6. Protocol-level benchmarks, AHA regional mapping, and recurrent design choices

The 2019 AHA work does not release a standalone “AHA!” dataset. Instead, it defines an Omniglot Extended Benchmark as a superset of the standard one-shot character-classification benchmark. The benchmark has two task variants, “oneshot” and “instance.” In oneshot, each run selects 20 distinct character classes from one evaluation alphabet, presents one train exemplar per class, and then presents 20 test exemplars written by different persons; matching uses nearest neighbor under minimum MSE in an internal representational state. In instance, the model must retrieve the exact exemplar among 20 distractor exemplars from the same character class, emphasizing pattern separation. The protocol uses Omniglot background alphabets for pre-training and 10 evaluation alphabets for testing, sweeps occlusion and noise over 10 levels up to just below 98%, and repeats each condition over 10 random seeds (Kowadlo et al., 2019).

This benchmark is explicitly unlabeled from the model’s training perspective: PS(DG) outputs act as internally generated labels for PR(EC-CA3), and evaluation uses minimum MSE matching and recall-loss rather than a trained classifier. On the standard oneshot setting with no corruption, the paper reports 86.4% accuracy for LTM+AHA using PR(EC-CA3) for matching, compared with 71.6% for the LTM-only baseline and 81.9% for the LTM+FastNN baseline (Kowadlo et al., 2019).

At the opposite end of the domain spectrum, the cardiac imaging paper introduces a route to build AHA-labeled datasets from CAMUS echocardiography through automatic mapping to the AHA 17-segment clinical standard. CAMUS contains 500 patients, apical 2-chamber and 4-chamber views, approximately 20 frames per sequence, and an official 400/50/50 train/val/test split, with expert masks for LV endocardium, LV epicardium, and left atrium at ED and ES. A graph-based landmark model with implicit anatomical correspondences is temporally refined without additional per-frame labels by penalizing velocity and acceleration discontinuities across sequences (Montalvo-García et al., 30 Jun 2026).

The regional mapping is atlas-based. For a point $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$3, the normalized long-axis coordinate is

$D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$4

with thresholds $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$5, $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$6, and $D^{align} = \{(a_i, q_i, r_i^+, r_i^-)\}$7, yielding basal, mid, apical, and apex-cap assignments. In 4CH views, basal and mid levels contribute inferoseptal versus anterolateral walls and the apical level contributes septal versus lateral walls; in 2CH views, basal and mid levels contribute inferior versus anterior walls and the apical level contributes inferior versus anterior walls. The code is publicly available, but the paper does not state a license or release precomputed AHA-labeled datasets; instead, it provides the ingredients to generate them from CAMUS (Montalvo-García et al., 30 Jun 2026).

Taken together, these cases clarify the breadth of “Aha! Datasets.” Some AHA resources are direct training sets; some are diagnostic benchmarks; some are synthetic failure corpora; some are structured prompt-and-annotation systems; some are protocol-level evaluations over existing data; and some are derived regional labels added to clinical image sequences. Several papers state that license terms are not specified in the paper text and defer practical reuse to project pages or repositories. A recurring design pattern is the use of structured perturbation or structured prompting—counterfactual hard negatives in audio, YAML perturbations in robotics, numeric perturbations in chain-of-thought, second-person vignette matrices in harms analysis, or title-templated task prompts in streaming video—to make latent reasoning or failure modes directly measurable.