MagicData340K: Annotated Text-to-Image Artifacts

Updated 10 July 2026

MagicData340K is a large-scale dataset that provides human-annotated text-image pairs with fine-grained artifact labels for evaluating text-to-image generation quality.
It introduces a hierarchical, multi-label taxonomy for classifying physical artifacts, using a binary Normal/Artifact split along with detailed L2 and L3 labels to capture diverse failure modes.
The dataset underpins MagicAssessor training, with advanced sampling and resampling strategies that lead to superior artifact recognition compared to general vision-language models.

Searching arXiv for MagicData340K and related papers to ground the article. MagicData340K is a large-scale dataset of generated text-image pairs for fine-grained artifact assessment in text-to-image generation. It was introduced as the data foundation of the broader MagicMirror framework, in which MagicData340K provides human supervision, MagicAssessor is the trained vision-language evaluator, and MagicBench is the benchmark enabled by that evaluator. The dataset is positioned as the first human-annotated large-scale dataset of generated images with fine-grained artifact labels, and its retained size is reported as 343,269 annotated text-image pairs, making the “340K” designation a rounded name rather than an exact count (Wang et al., 12 Sep 2025).

1. Conceptual role and problem setting

MagicData340K was created to address a specific deficiency in evaluation of contemporary text-to-image systems: strong progress in instruction following, aesthetics, and generic image quality has not eliminated physical artifacts such as malformed hands, distorted faces, irrational overlaps, impossible spatial arrangements, and object shape errors (Wang et al., 12 Sep 2025). The dataset is therefore not a general image-classification resource, not an aesthetic-preference corpus, and not a semantic alignment benchmark. Its target problem is physically grounded artifact assessment: deciding whether a generated image is normal or artifact-bearing, and, if artifact-bearing, specifying which failure modes are present.

Within MagicMirror, the dataset serves two linked functions. First, it is the principal training corpus for MagicAssessor, a vision-LLM trained to output detailed artifact judgments and hierarchical labels. Second, its taxonomy defines the conceptual basis for MagicBench, the automated benchmark that scores current text-to-image models by artifact category. In that sense, MagicData340K is both a supervised dataset and a specification of the label space used for downstream automated evaluation (Wang et al., 12 Sep 2025).

The dataset is explicitly multi-label beyond the top level. The paper states that top-level assessment is binary—Normal versus Artifact—but that images may contain multiple artifact types, so annotators can assign multiple labels at the higher levels of the taxonomy. This design reflects the paper’s view that generated-image defects are structurally heterogeneous and cannot be reduced to a single plausibility score or undifferentiated artifact region (Wang et al., 12 Sep 2025).

2. Data construction, sources, and corpus composition

Dataset construction begins with prompt curation. The authors report 50,000 prompts drawn from three sources: 23,000 user prompts sampled from Pick-a-Pic, 23,000 prompts generated by GPT by combining entities and artistic styles from a curated database, and 4,000 prompts specifically targeting human subjects, also composed by GPT using human-attribute resources (Wang et al., 12 Sep 2025). The paper additionally mentions a large-scale database of entities, artistic styles, and human attributes from a wide range of sources, but does not enumerate those sources in the provided text.

Images were generated by multiple text-to-image systems: FLUX.1-dev / FLUX.1-schnell, Kolors1.0, SD3.5, SD3, Midjourney-v6.1, and an internal model (Wang et al., 12 Sep 2025). The dataset is therefore intentionally heterogeneous across generators, prompt sources, artistic styles, and subject domains. All images are generated images; the corpus is not a mixed real/synthetic dataset.

After annotation, the authors filtered out inappropriate content and retained 343,269 text-image pairs. The appendix provides explicit split sizes, while also introducing an unresolved inconsistency for the chain-of-thought subset.

Split/statistic	Size
Total	343,269
Train	325,238
Test	17,366
CoT set	1,294
CoT set in main-text table	1,835

The discrepancy between 1,294 and 1,835 is internal to the paper and is not resolved in the supplied text. The appendix prose and experimental setup state that the CoT dataset contains 1,294 examples, whereas the main-text table labeled detail_data_statistics shows 1,835 in the CoT column and total row. This should be treated as unresolved until checked against the official release (Wang et al., 12 Sep 2025).

The dataset is also nearly balanced at the binary level. The appendix gives 173,768 Normal images and 169,501 Artifacts images. This near-even split applies only to the top level; the finer label hierarchy is strongly imbalanced, with some artifact classes dominating and others extremely rare (Wang et al., 12 Sep 2025).

3. Hierarchical label taxonomy

MagicData340K introduces a hierarchical multi-label taxonomy with three levels. At L1, the label is binary: Normal or Artifact. At L2, the dataset defines major artifact families. At L3, many L2 families are refined into more specific subcategories. The paper states explicitly that L1 is a simple binary choice, whereas multiple L2 and L3 labels may be assigned to a single image (Wang et al., 12 Sep 2025).

Operationally, the taxonomy includes five L2 categories plus a residual class:

Irrational Element Attributes
Irrational Element Interaction
Abnormal Human Anatomy
Abnormal Animal Anatomy
Abnormal Object Morphology
Other Irrationalities

These categories are motivated by three broader conceptual groups named in the paper as object anatomy, attribute, and interaction. The resulting taxonomy is intended to cover subject anatomy, visual attributes, and interactions between elements or subjects (Wang et al., 12 Sep 2025).

The paper defines the L2 categories and their L3 refinements in detail. Irrational Element Attributes contains Abnormal Material Texture, Abnormal Detail Drawing, Abnormal Element Proportion, and Abnormal Color Combination. Irrational Element Interaction contains Abnormal Light and Shadow Effect, Abnormal Element Overlap, and Abnormal Spatial Position. Abnormal Human Anatomy contains Limb Structure Deformity, Trunk Structure Deformity, Hand Structure Deformity, Foot Structure Deformity, Facial Structure Deformity, Abnormal Human Anatomy, and Abnormal and Uncoordinated Posture. Abnormal Animal Anatomy contains Abnormal Limb Structure, Abnormal Posture Presentation, and Abnormal Head Structure. Abnormal Object Morphology does not list L3 subcategories in the appendix taxonomy (Wang et al., 12 Sep 2025).

Several annotation rules are central to interpreting these labels. Annotators mark only the two most obvious issues, except in specified cases involving many people or a single person with more than three abnormalities, where a generic Abnormal Human Anatomy L3 label may be used. Only obvious abnormalities should be marked; if an issue cannot be identified within 3 seconds, may have a reasonable explanation, or belongs to an imaginable special case, it is not considered abnormal. A strong AI-generated visual feeling is not itself a structural abnormality. Text errors in the image are ignored. The prompt may be used to interpret whether apparently unrealistic content should nonetheless be labeled normal because it matches the intended style or subject matter (Wang et al., 12 Sep 2025).

A plausible implication is that the dataset is deliberately conservative with respect to subtle defects and style-centric judgments. Its supervision is oriented toward structural and physical plausibility rather than toward generic synthetic-image oddness.

4. Annotation process and statistical profile

The dataset was manually annotated by humans, but the exact number of annotators is not reported in the supplied text. The paper describes a pilot study with expert annotators, iterative refinement of guidelines and interface, and continuous expert oversight to ensure that rules were followed (Wang et al., 12 Sep 2025). It also mentions an intuitive annotation interface shown in the appendix. However, the text does not provide the number of annotators per image, inter-annotator agreement, adjudication procedures, majority-vote details, annotation time, or annotation cost.

The distribution over artifact classes is notably long-tailed. Among the 169,501 artifact images, the appendix reports the following L2 counts:

L2 category	Count	Share of artifact images
Abnormal Human Anatomy	104,289	61.53%
Irrational Element Interaction	61,994	36.6%
Abnormal Object Morphology	37,085	21.88%
Abnormal Animal Anatomy	23,464	13.84%
Irrational Element Attributes	881	0.52%
Other Irrationalities	20	0.01%

Because the labeling is multi-label, these percentages sum to more than 100% (Wang et al., 12 Sep 2025).

At L3, the imbalance becomes more pronounced. Hand Structure Deformity alone accounts for 54,114 cases, and the generic Abnormal Human Anatomy L3 label accounts for 42,327. By contrast, some attribute-level subcategories are rare: Abnormal Material Texture has 54 cases, Abnormal Color Combination 23, and Abnormal Light and Shadow Effect 258 (Wang et al., 12 Sep 2025). The paper explicitly identifies Abnormal Human Anatomy, especially Hand Structure Deformity, as the dominant class, while Irrational Element Attributes and several L3 labels are extremely rare.

This long-tail structure is not merely descriptive; it conditions how the dataset can be used. The paper ties the imbalance directly to its training strategies for MagicAssessor, including Multi-Bucket Sampling and positive data resampling for difficult-but-correct cases such as anatomically correct hands. A plausible implication is that naive empirical-risk minimization on the raw class distribution would bias a model toward overpredicting common anatomy failures while neglecting rarer interaction or attribute errors.

5. Use in MagicAssessor training

MagicData340K is used in a two-stage training pipeline for MagicAssessor, whose base model is Qwen2.5-VL-7B (Wang et al., 12 Sep 2025). Stage 1 is supervised fine-tuning on a curated CoT subset. These examples are constructed by selecting high-quality annotated samples, having humans write detailed textual descriptions for each applied label, and then feeding the prompt, image, labels, and descriptions to GPT-4o, which produces a structured chain of thought. The prompting enforces a four-step reasoning structure: prompt review, image description, element examination, and conclusion.

Stage 2 uses Group Relative Policy Optimization on the main resampled dataset. The dataset’s hierarchical labels define the reward structure. The paper gives the final reward as

$R = r_c \cdot \sum_{l=0}^{n} 2^{n-l} r_l,\quad n=3$

where $r_c \in \{0,1\}$ is the consistency reward, $r_0 \in \{0,1\}$ is the format/parsing reward, $r_1 \in \{0,1\}$ is the L1 reward, and $r_2, r_3 \in [0,1]$ are the L2/L3 rewards (Wang et al., 12 Sep 2025). For multi-label L2/L3 scoring, the appendix defines

$r_{2,3} = \operatorname{clamp}(0.6 \cdot n_{correct} - 0.3 \cdot (n_{miss} + n_{extra}), \min=0, \max=1).$

This explicitly rewards correct multi-label predictions while penalizing both missing labels and extra labels.

The output format learned from the dataset combines free-form reasoning and structured prediction. The model is trained to produce natural-language reasoning in > ... followed by a boxed answer with hierarchical labels. Example outputs include {"Whether Normal": True} for normal images and structured label sets for artifact images (Wang et al., 12 Sep 2025).

Because of the dataset’s imbalance, the paper introduces Multi-Bucket Sampling. Each training batch is sampled from five buckets—normal images and four main L2 artifact labels—in a 4:1:1:1:1 ratio. The remaining two rare L2 labels are omitted from this strategy because together they account for less than 1% of the dataset (Wang et al., 12 Sep 2025). The authors also perform positive data resampling for hard correct cases, particularly correct hands, to avoid degenerate behavior such as learning that any hand implies an artifact.

The empirical evidence reported in the paper is dataset-specific rather than purely methodological. Without Multi-Bucket Sampling, the model overfits to Abnormal Human Anatomy, and minority-class recall collapses; for Irrational Element Interaction, the appendix reports recall dropping to 0.0097 and F1 to 0.0185. Without positive data resampling, performance on distinguishing correct hands from malformed ones deteriorates. Without the consistency reward, metrics drop and explanations become more disorganized (Wang et al., 12 Sep 2025).

6. Benchmark role, performance implications, and scope boundaries

MagicData340K is not itself the evaluation set for MagicBench, but it is the resource that makes MagicBench possible. The benchmark is constructed by creating 800 prompts spanning human, animal, and object scenarios, generating one image per prompt for each evaluated text-to-image model, verifying that the required subject is present, running MagicAssessor to predict artifact labels, and computing category scores (Wang et al., 12 Sep 2025). The benchmark score is defined as

$Score_{label}=100\cdot\left(1-\frac{N_{label}}{N_{label\_set}}\right),$

where $N_{label}$ is the number of images assigned a given L2 label or the top-level artifact label, and $N_{label\_set}$ is 300 for Human, 200 for Animal, 300 for Object, and 800 for Interaction and Artifacts (Wang et al., 12 Sep 2025). The overall score equals $Score_{artifacts}$ , that is, the percentage of images judged normal at the binary level.

The main reported evidence for the dataset’s value is that models trained on it outperform general-purpose VLM baselines on artifact recognition. MagicAssessor-7B achieves L2 Macro F1 of 0.5261, L2 Micro F1 of 0.5580, and Artifacts F1 of 0.7001. The paper compares this with GPT-4o Artifacts F1 of 0.4217 and Gemini-2.5-pro Artifacts F1 of 0.6560, while stating that general open-source VLMs perform substantially worse (Wang et al., 12 Sep 2025). These numbers support the claim that MagicData340K supplies supervision that is both fine-grained and operationally useful.

Several caveats delimit the dataset’s scope. Class imbalance is severe, some labels are extremely rare, the “two most obvious issues” rule means that heavily flawed images may not receive exhaustive label sets, and the 3-second obviousness criterion may exclude subtle artifacts (Wang et al., 12 Sep 2025). Style-driven “AI-generated feeling” is explicitly excluded when no structural issue exists, so the dataset is not a general benchmark for synthetic-image realism. The supplied text also does not clearly state whether the raw dataset is publicly downloadable, under what license it is released, or what exact file format it uses; only the project page is explicitly given (Wang et al., 12 Sep 2025).

A further boundary concerns nomenclature. Despite superficial name overlap, the coreset-selection paper “MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-LLMs” does not explicitly define any dataset named MagicData340K; it introduces MAGIC as a selection method and evaluates only subsets such as 20% of LLaVA-665K and 20% of Vision-Flan-186K (Biswas et al., 25 May 2026). Likewise, MathWriting is a dataset for handwritten mathematical expression recognition and is unrelated in purpose, modality, and annotation scheme (Gervais et al., 2024). These distinctions matter because MagicData340K is specifically a generated-image artifact dataset for text-to-image assessment, not a generic multimodal subset and not a handwriting-recognition corpus.