---
title: 'AttriData: LLM Error Attribution Dataset'
url: https://www.emergentmind.com/topics/attridata
type: topic
---

# AttriData: LLM Error Attribution Dataset

Searching arXiv for the cited works to ground the article and confirm the most relevant "AttriData" references.
AttriData is a manually annotated evaluation dataset for error attribution in large language model judging. It was introduced together with a Misattribution Framework and used to train MisAttributionLLM, which is presented as the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback [2507.08459]. Its defining objective is not merely to determine whether an answer is good or bad, but to diagnose why a flawed answer failed, using a structured taxonomy of failure modes and a standardized three-part output format.

## 1. Definition and scope

AttriData was created for a setting in which mainstream LLM platforms generate massive user-model interactions daily and developers need automated diagnosis rather than scalar evaluation alone [2507.08459]. In this formulation, each instance contains a question, a reference answer, a model answer, a human-assigned score, a misattribution label or `NULL`, and feedback. The target output is organized in three lines: evaluation reason, error attribution, and a score from 0 to 3.

The scoring scheme is semantically defined. A score of 3 denotes a correct answer with no errors; 2 denotes a partially correct answer; 1 denotes a completely incorrect answer; and 0 denotes an off-topic response or safety violation. Misattribution is assigned only when the score is less than 3; if the score is 3, the misattribution field is `NULL` [2507.08459]. The dataset therefore supports a joint task combining ordinal assessment, categorical diagnosis, and free-text explanation.

The corpus contains 21,702 samples, with 18,806 in the training set and 2,896 in the test set. Of these, 8,026 samples contain non-`NULL` misattribution labels. The language distribution is majority Chinese, with 1,321 English samples. The questions cover six broad domains: NLP Basic, Math, Reasoning, Text Generation, Question and Answer, and Professional Field [2507.08459].

## 2. Misattribution framework

The Misattribution Framework defines 6 primary categories and 15 secondary categories. It is designed around single-label attribution: even when an answer could plausibly exhibit multiple faults, the annotation protocol selects the most critical misattribution category [2507.08459].

| Primary category | Secondary categories | Focus |
|---|---|---|
| Instruction Following | Content Inconsistency, Format Inconsistency, Length Inconsistency, Truncation | Failure to obey prompt constraints |
| Response Quality | Duplicate, Refusal to Answer, Missing Answers, Noisy, Typo | Surface quality and completeness |
| Knowledge Ability | Hallucination, Incorrect Answers | Factual and objective-answer errors |
| Reasoning Capability | Process Error, Result Error | Inferential failures |
| Safety | Safety | Harmful or risky outputs |
| Other Errors | Others | Residual uncaptured cases |

Within this scheme, “Content Inconsistency” is defined as text that fails to meet required content standards such as language, structure, theme, and style, whereas “Format Inconsistency” covers responses that do not conform to explicit formatting constraints. “Hallucination” denotes generated content inconsistent with real-world facts or the user’s input, while “Incorrect Answers” is reserved for objective questions whose answers are simply wrong. “Process Error” refers to logical flaws in reasoning, and “Result Error” refers to errors in final outcomes, particularly in mathematics and coding [2507.08459].

The framework is operational rather than merely descriptive. It is used to supervise a judge model that must decide not only whether an answer is deficient, but which failure type is most relevant. A common misconception is that this kind of dataset is only an extension of scoring with richer feedback. The task design shows otherwise: misattribution is treated as a separate supervised signal, and the paper’s ablations indicate that removing this signal degrades both diagnosis and score alignment [2507.08459].

## 3. Dataset construction and annotation protocol

AttriData was constructed over a three-month process. Question instances were drawn from Xie et al. (2023) and further refined and generalized following the methodology of Lin et al. (2024). Candidate model answers were generated using ERNIE Bot and Hunyuan. For feedback, the authors used GPT-4 to generate initial feedback text, but all such feedback was subjected to rigorous human verification and manually corrected when necessary [2507.08459].

The annotation pipeline begins with guideline training for scoring, misattribution, and feedback. Each sample is independently annotated by 3 annotators and then reviewed by 1 senior expert. If the annotators disagree, the senior expert makes the final decision. The annotator pool comprises 36 annotators and 12 senior annotation experts; the senior group includes 5 domain experts focused on reasoning error detection. Quality control is batch-based: the dataset is divided into 20 batches, and 30% of submissions in each batch are randomly checked. If accuracy falls below 98%, the batch is returned for re-annotation [2507.08459].

Inter-annotator agreement is reported via Fleiss’ kappa: 0.875 for scores and 0.832 for misattribution labels, which the paper interprets as almost perfect agreement. This matters because error attribution is inherently more subjective than scalar scoring, and the agreement numbers indicate that the framework is stable enough to support supervised learning [2507.08459].

A further design choice is the prevalence of satisfactory answers. Since only 8,026 of 21,702 samples carry non-`NULL` misattribution, the dataset is intentionally skewed toward acceptable responses. This suggests a deployment-oriented distribution rather than a benchmark saturated with failures. A plausible implication is that AttriData is optimized for realistic evaluation pipelines, where high precision in identifying true failures is as important as broad coverage of failure types.

## 4. Modeling and evaluation

AttriData is used to fine-tune Qwen2.5-7B into MisAttributionLLM. The training setup uses 8 × 40G A100 GPUs, DeepSpeed ZeRO stage 3, AdamW, weight decay 0.1, learning rate \(1.0\times10^{-4}\), warmup ratio 10%, batch size 16, and 2 epochs [2507.08459]. The model is trained in a standard supervised generative format: it sequentially generates feedback, then misattribution, then score.

On the AttriData test set, MisAttributionLLM achieves Pearson 0.935, Spearman 0.946, and Kendall-Tau 0.934 for alignment with human scores. In the same evaluation, GPT-4 attains 0.817, 0.843, and 0.799, and the untuned Qwen2.5-7B base model attains 0.461, 0.486, and 0.461 [2507.08459]. For binary misattribution detection, MisAttributionLLM reaches precision 0.985, recall 0.956, and F1 0.970; for multi-class misattribution classification, it reaches accuracy 0.836 and micro-F1 0.829 [2507.08459].

The ablations are especially revealing. When misattribution supervision is removed, detection performance drops to precision 0.348, recall 0.998, and F1 0.516, indicating near-universal over-flagging. Score correlation also degrades to Pearson 0.838, Spearman 0.917, and Kendall-Tau 0.875 [2507.08459]. This demonstrates that structured diagnosis is not merely auxiliary; it improves scalar evaluation as well.

Feedback quality is evaluated by pairwise human preference on 949 misattributed test samples. MisAttributionLLM is preferred over GPT-4 in 60.41% of cases and over GPT-3.5 in 85.20% of cases [2507.08459]. Off-benchmark generalization is assessed on AlignBench, where MisAttributionLLM obtains Pearson 0.779, Spearman 0.803, and Kendall-Tau 0.759, close to GPT-4’s 0.786, 0.792, and 0.767 [2507.08459]. These results position AttriData not only as a dataset for benchmark optimization but also as a training substrate for deployable judge models.

## 5. Relation to broader attribute-centric data design

Although AttriData is an LLM evaluation dataset rather than a visual or multimodal benchmark, it belongs to a broader family of attribute-centric resources that make latent structure explicit. In visual storytelling, AttriStory frames fine-grained attribute realization as a distinct benchmark axis, with 200 stories, 5 scenes per story, 1,000 scene descriptions, 10 artistic styles, and explicit positive and negative attribute-object pairs such as \((\text{“pink”}, \text{“dress”})\) and \((\text{“pink”}, \text{“lilies”})\) [2605.20777]. In fashion recognition, iFashion-Attribute provides 1,012,947 training images, 9,897 validation images, and 39,706 test images labeled with 228 fine-grained attributes across 8 groups, making group-structured multi-label prediction explicit at million scale [1906.05750].

A similar pattern appears in open-vocabulary detection and synthetic data generation. Objects365-Attr extends Objects365 with 5.6M object-level attribute descriptions across 1.4M bounding boxes and organizes attributes into color, material, state, texture, and tone [2409.06300]. AttrSyn uses GPT-4 to generate attribute concepts and values, producing 25,000 diversity configurations and 6,000 synthetic images per dataset for zero-shot fine-grained bird classification [2504.04510]. These resources differ in modality and task, but they share a design principle with AttriData: they replace underspecified labels with structured attribute-level supervision.

The same logic extends to generation and biomedical annotation. Att-Adapter learns continuous multi-attribute control from image-text-attribute vectors without paired synthetic edits, while AttriGen uses a dual-model pipeline to enrich blood-cell datasets with an 8-class cell type and 11 morphological attributes, reporting 94.62% Global Average Accuracy on WBCAtt [2503.11937; 2509.26185]. This suggests that AttriData is part of a wider methodological shift toward datasets that expose failure modes, compositional bindings, or domain attributes as first-class supervision rather than latent side effects.

## 6. Limitations and significance

The paper identifies three explicit limitations. First, feedback quality is partly bounded by GPT-4 because GPT-4 generated the initial feedback before human verification. Second, the taxonomy is not exhaustive, which is why an “Others” category is retained. Third, attribution is single-label: the framework identifies one most critical misattribution category rather than modeling multiple simultaneous errors [2507.08459].

These constraints define the current scope of AttriData. It is not a multi-label causal ontology of all answer defects, and it does not claim complete coverage of the error space of large language models. It is better understood as a supervised interface for practical failure diagnosis. In that role, its contribution is substantial: it operationalizes error attribution as a learnable task, shows that diagnosis improves scoring, and provides a high-agreement, quality-controlled corpus for training judge models [2507.08459].

In the broader history of attribute-centric datasets, AttriData is notable because it relocates the attribute idea from perceptual semantics to evaluation semantics. Instead of encoding color, material, or morphology, it encodes structured failure types in model outputs. This suggests a general principle: once a domain’s important latent distinctions are made explicit and consistently annotated, they can support not only better analysis but also better models.

Source: https://www.emergentmind.com/topics/attridata