AttriData: LLM Error Attribution Dataset
- AttriData is a manually annotated evaluation dataset for error attribution in large language models, defining a structured taxonomy for diagnosing flawed answers.
- It integrates ordinal scoring, categorical misattribution, and free-text feedback to offer a comprehensive analysis of LLM performance with high inter-annotator agreement.
- The dataset underpins the training of MisAttributionLLM, which achieves strong metrics (e.g., Pearson 0.935, F1 0.970) for both error detection and score alignment.
Searching arXiv for the cited works to ground the article and confirm the most relevant "AttriData" references. AttriData is a manually annotated evaluation dataset for error attribution in LLM judging. It was introduced together with a Misattribution Framework and used to train MisAttributionLLM, which is presented as the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback (Xu et al., 11 Jul 2025). Its defining objective is not merely to determine whether an answer is good or bad, but to diagnose why a flawed answer failed, using a structured taxonomy of failure modes and a standardized three-part output format.
1. Definition and scope
AttriData was created for a setting in which mainstream LLM platforms generate massive user-model interactions daily and developers need automated diagnosis rather than scalar evaluation alone (Xu et al., 11 Jul 2025). In this formulation, each instance contains a question, a reference answer, a model answer, a human-assigned score, a misattribution label or NULL, and feedback. The target output is organized in three lines: evaluation reason, error attribution, and a score from 0 to 3.
The scoring scheme is semantically defined. A score of 3 denotes a correct answer with no errors; 2 denotes a partially correct answer; 1 denotes a completely incorrect answer; and 0 denotes an off-topic response or safety violation. Misattribution is assigned only when the score is less than 3; if the score is 3, the misattribution field is NULL (Xu et al., 11 Jul 2025). The dataset therefore supports a joint task combining ordinal assessment, categorical diagnosis, and free-text explanation.
The corpus contains 21,702 samples, with 18,806 in the training set and 2,896 in the test set. Of these, 8,026 samples contain non-NULL misattribution labels. The language distribution is majority Chinese, with 1,321 English samples. The questions cover six broad domains: NLP Basic, Math, Reasoning, Text Generation, Question and Answer, and Professional Field (Xu et al., 11 Jul 2025).
2. Misattribution framework
The Misattribution Framework defines 6 primary categories and 15 secondary categories. It is designed around single-label attribution: even when an answer could plausibly exhibit multiple faults, the annotation protocol selects the most critical misattribution category (Xu et al., 11 Jul 2025).
| Primary category | Secondary categories | Focus |
|---|---|---|
| Instruction Following | Content Inconsistency, Format Inconsistency, Length Inconsistency, Truncation | Failure to obey prompt constraints |
| Response Quality | Duplicate, Refusal to Answer, Missing Answers, Noisy, Typo | Surface quality and completeness |
| Knowledge Ability | Hallucination, Incorrect Answers | Factual and objective-answer errors |
| Reasoning Capability | Process Error, Result Error | Inferential failures |
| Safety | Safety | Harmful or risky outputs |
| Other Errors | Others | Residual uncaptured cases |
Within this scheme, “Content Inconsistency” is defined as text that fails to meet required content standards such as language, structure, theme, and style, whereas “Format Inconsistency” covers responses that do not conform to explicit formatting constraints. “Hallucination” denotes generated content inconsistent with real-world facts or the user’s input, while “Incorrect Answers” is reserved for objective questions whose answers are simply wrong. “Process Error” refers to logical flaws in reasoning, and “Result Error” refers to errors in final outcomes, particularly in mathematics and coding (Xu et al., 11 Jul 2025).
The framework is operational rather than merely descriptive. It is used to supervise a judge model that must decide not only whether an answer is deficient, but which failure type is most relevant. A common misconception is that this kind of dataset is only an extension of scoring with richer feedback. The task design shows otherwise: misattribution is treated as a separate supervised signal, and the paper’s ablations indicate that removing this signal degrades both diagnosis and score alignment (Xu et al., 11 Jul 2025).
3. Dataset construction and annotation protocol
AttriData was constructed over a three-month process. Question instances were drawn from Xie et al. (2023) and further refined and generalized following the methodology of Lin et al. (2024). Candidate model answers were generated using ERNIE Bot and Hunyuan. For feedback, the authors used GPT-4 to generate initial feedback text, but all such feedback was subjected to rigorous human verification and manually corrected when necessary (Xu et al., 11 Jul 2025).
The annotation pipeline begins with guideline training for scoring, misattribution, and feedback. Each sample is independently annotated by 3 annotators and then reviewed by 1 senior expert. If the annotators disagree, the senior expert makes the final decision. The annotator pool comprises 36 annotators and 12 senior annotation experts; the senior group includes 5 domain experts focused on reasoning error detection. Quality control is batch-based: the dataset is divided into 20 batches, and 30% of submissions in each batch are randomly checked. If accuracy falls below 98%, the batch is returned for re-annotation (Xu et al., 11 Jul 2025).
Inter-annotator agreement is reported via Fleiss’ kappa: 0.875 for scores and 0.832 for misattribution labels, which the paper interprets as almost perfect agreement. This matters because error attribution is inherently more subjective than scalar scoring, and the agreement numbers indicate that the framework is stable enough to support supervised learning (Xu et al., 11 Jul 2025).
A further design choice is the prevalence of satisfactory answers. Since only 8,026 of 21,702 samples carry non-NULL misattribution, the dataset is intentionally skewed toward acceptable responses. This suggests a deployment-oriented distribution rather than a benchmark saturated with failures. A plausible implication is that AttriData is optimized for realistic evaluation pipelines, where high precision in identifying true failures is as important as broad coverage of failure types.
4. Modeling and evaluation
AttriData is used to fine-tune Qwen2.5-7B into MisAttributionLLM. The training setup uses 8 × 40G A100 GPUs, DeepSpeed ZeRO stage 3, AdamW, weight decay 0.1, learning rate , warmup ratio 10%, batch size 16, and 2 epochs (Xu et al., 11 Jul 2025). The model is trained in a standard supervised generative format: it sequentially generates feedback, then misattribution, then score.
On the AttriData test set, MisAttributionLLM achieves Pearson 0.935, Spearman 0.946, and Kendall-Tau 0.934 for alignment with human scores. In the same evaluation, GPT-4 attains 0.817, 0.843, and 0.799, and the untuned Qwen2.5-7B base model attains 0.461, 0.486, and 0.461 (Xu et al., 11 Jul 2025). For binary misattribution detection, MisAttributionLLM reaches precision 0.985, recall 0.956, and F1 0.970; for multi-class misattribution classification, it reaches accuracy 0.836 and micro-F1 0.829 (Xu et al., 11 Jul 2025).
The ablations are especially revealing. When misattribution supervision is removed, detection performance drops to precision 0.348, recall 0.998, and F1 0.516, indicating near-universal over-flagging. Score correlation also degrades to Pearson 0.838, Spearman 0.917, and Kendall-Tau 0.875 (Xu et al., 11 Jul 2025). This demonstrates that structured diagnosis is not merely auxiliary; it improves scalar evaluation as well.
Feedback quality is evaluated by pairwise human preference on 949 misattributed test samples. MisAttributionLLM is preferred over GPT-4 in 60.41% of cases and over GPT-3.5 in 85.20% of cases (Xu et al., 11 Jul 2025). Off-benchmark generalization is assessed on AlignBench, where MisAttributionLLM obtains Pearson 0.779, Spearman 0.803, and Kendall-Tau 0.759, close to GPT-4’s 0.786, 0.792, and 0.767 (Xu et al., 11 Jul 2025). These results position AttriData not only as a dataset for benchmark optimization but also as a training substrate for deployable judge models.
5. Relation to broader attribute-centric data design
Although AttriData is an LLM evaluation dataset rather than a visual or multimodal benchmark, it belongs to a broader family of attribute-centric resources that make latent structure explicit. In visual storytelling, AttriStory frames fine-grained attribute realization as a distinct benchmark axis, with 200 stories, 5 scenes per story, 1,000 scene descriptions, 10 artistic styles, and explicit positive and negative attribute-object pairs such as and (Sreenivas et al., 20 May 2026). In fashion recognition, iFashion-Attribute provides 1,012,947 training images, 9,897 validation images, and 39,706 test images labeled with 228 fine-grained attributes across 8 groups, making group-structured multi-label prediction explicit at million scale (Guo et al., 2019).
A similar pattern appears in open-vocabulary detection and synthetic data generation. Objects365-Attr extends Objects365 with 5.6M object-level attribute descriptions across 1.4M bounding boxes and organizes attributes into color, material, state, texture, and tone (Qi et al., 2024). AttrSyn uses GPT-4 to generate attribute concepts and values, producing 25,000 diversity configurations and 6,000 synthetic images per dataset for zero-shot fine-grained bird classification (Wang et al., 6 Apr 2025). These resources differ in modality and task, but they share a design principle with AttriData: they replace underspecified labels with structured attribute-level supervision.
The same logic extends to generation and biomedical annotation. Att-Adapter learns continuous multi-attribute control from image-text-attribute vectors without paired synthetic edits, while AttriGen uses a dual-model pipeline to enrich blood-cell datasets with an 8-class cell type and 11 morphological attributes, reporting 94.62% Global Average Accuracy on WBCAtt (Cho et al., 15 Mar 2025, Houmaidi et al., 30 Sep 2025). This suggests that AttriData is part of a wider methodological shift toward datasets that expose failure modes, compositional bindings, or domain attributes as first-class supervision rather than latent side effects.
6. Limitations and significance
The paper identifies three explicit limitations. First, feedback quality is partly bounded by GPT-4 because GPT-4 generated the initial feedback before human verification. Second, the taxonomy is not exhaustive, which is why an “Others” category is retained. Third, attribution is single-label: the framework identifies one most critical misattribution category rather than modeling multiple simultaneous errors (Xu et al., 11 Jul 2025).
These constraints define the current scope of AttriData. It is not a multi-label causal ontology of all answer defects, and it does not claim complete coverage of the error space of LLMs. It is better understood as a supervised interface for practical failure diagnosis. In that role, its contribution is substantial: it operationalizes error attribution as a learnable task, shows that diagnosis improves scoring, and provides a high-agreement, quality-controlled corpus for training judge models (Xu et al., 11 Jul 2025).
In the broader history of attribute-centric datasets, AttriData is notable because it relocates the attribute idea from perceptual semantics to evaluation semantics. Instead of encoding color, material, or morphology, it encodes structured failure types in model outputs. This suggests a general principle: once a domain’s important latent distinctions are made explicit and consistently annotated, they can support not only better analysis but also better models.