MagicBench: Dual Evaluation Benchmarks

• MagicBench is a label used for two benchmarks: MagicMotion evaluates video trajectory control, while MagicMirror assesses fine-grained artifacts in text-to-image generation.
• MagicMotion uses 600 high-quality, 49-frame videos with dense segmentation and sparse bounding box annotations, evaluated via FVD, FID, Mask_IoU, and Box_IoU across six object-count categories.
• MagicMirror leverages 800 prompt-generated images processed by MagicAssessor to diagnose physical artifacts, stratifying errors in anatomy, morphology, interactions, and attributes.

Searching arXiv for "MagicBench" to identify the relevant papers and disambiguate usages. MagicBench is a benchmark name used for two distinct evaluation frameworks introduced in separate 2025 papers. In "MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance," MagicBench denotes a benchmark for trajectory-controllable video generation that evaluates both video quality and trajectory control accuracy under varying numbers of moving objects (Li et al., 20 Mar 2025). In "MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation," MagicBench denotes an automated benchmark for fine-grained assessment of physical artifacts in text-to-image generation (Wang et al., 12 Sep 2025). The term therefore requires contextual disambiguation, and it should not be confused with MAGIC, a benchmark for inter-context conflicts in retrieval-augmented generation (Lee et al., 29 Jul 2025).

1. Term, scope, and disambiguation

The reuse of the name "MagicBench" spans different modalities, evaluation targets, and methodological assumptions. One benchmark is centered on motion controllability in image-to-video synthesis; the other is centered on artifact diagnosis in text-to-image generation. Their overlap is nominal rather than procedural.

Usage of "MagicBench"	Modality	Primary assessment
MagicMotion	Trajectory-controllable video generation	FVD, FID, Mask_IoU, Box_IoU, stratified by number of moving objects
MagicMirror	Text-to-image generation	Fine-grained physical artifact assessment via subject verification and MagicAssessor

This naming ambiguity matters because the two benchmarks operationalize model quality differently. In the video setting, the emphasis is controllable motion under dense-to-sparse trajectory conditions; in the text-to-image setting, the emphasis is taxonomic diagnosis of physical artifacts rather than preference or aesthetics alone (Li et al., 20 Mar 2025, Wang et al., 12 Sep 2025).

2. MagicBench in trajectory-controllable video generation

Within MagicMotion, MagicBench was introduced to address a gap in the evaluation of trajectory-controllable video generation. The motivating claim is that existing datasets and benchmarks are insufficient because most focus on small or limited datasets such as DAVIS and VIPSeg, or are private, and because prior work neglects the significant impact that the number of moving objects has on controllability and video generation performance (Li et al., 20 Mar 2025).

The benchmark contains 600 videos with trajectory annotations. Videos are sampled to 49 frames each and resized to 720p (480×720) for evaluation. Each video is accompanied by dense segmentation mask and sparse bounding box trajectory annotations. The dataset is categorized into 6 groups based on the number of controlled foreground objects—1, 2, 3, 4, 5, and more than 5 objects—with 100 high-quality videos in each category (Li et al., 20 Mar 2025).

MagicBench supports three control formats. Segmentation masks provide pixel-wise object trajectories throughout the video. Bounding boxes provide object positions and extents at each frame. Sparse boxes provide bounding box annotations for only a few frames, specifically less than 10 per video. This design aligns the benchmark with dense-to-sparse control scenarios and permits evaluation of both precise and user-friendly forms of motion conditioning (Li et al., 20 Mar 2025).

The benchmarking protocol is straightforward: the model receives the initial frame, prompt, and the relevant trajectory annotation; it then produces a 49-frame video that is compared with ground truth on visual and control metrics. Metrics are reported not only in aggregate but also per object category, so that multi-object scaling behavior becomes explicit rather than being averaged away (Li et al., 20 Mar 2025).

3. Video-side evaluation protocol and empirical use

MagicBench in the video setting evaluates two dimensions: video/image quality and trajectory control accuracy. For visual quality, it uses FVD (Fréchet Video Distance, $\downarrow$) and FID (Fréchet Inception Distance, $\downarrow$). For controllability, it uses Mask_IoU ($\uparrow$) and Box_IoU ($\uparrow$) (Li et al., 20 Mar 2025).

The mask-based control metric is defined per object and frame as

$$\text{IoU}_\text{mask}(i, t) = \frac{|M_{gen}(i, t) \cap M_{gt}(i, t)|}{|M_{gen}(i, t) \cup M_{gt}(i, t)|},$$

where $M_{gen}(i,t)$ is the generated mask for object $i$ at frame $t$, obtained via SAM2 using the ground-truth mask from the first frame, and $M_{gt}(i,t)$ is the ground-truth mask. Mask_IoU is then averaged over objects and frames. The box-based metric is analogously defined as

$$\text{IoU}_\text{box}(i, t) = \frac{|B_{gen}(i, t) \cap B_{gt}(i, t)|}{|B_{gen}(i, t) \cup B_{gt}(i, t)|}.$$

For methods not supporting long videos, uniform frame-sampling is applied, and for methods not using masks directly, masks, boxes, or points are extracted according to each method’s supported input format (Li et al., 20 Mar 2025).

The benchmark was used to report that MagicMotion outperforms previous methods across all metrics on MagicBench and DAVIS. A quantitative example given for MagicMotion-Stage1 under mask conditions is FID = 87.13, FVD = 112.69, Mask_IoU = 91.57\%, and Box_IoU = 87.75\%. The paper further reports that competing methods degrade rapidly in trajectory control and video quality as object count increases, whereas MagicMotion maintains high scores, and that MagicBench makes such scalability differences measurable because results are stratified by the number of moving objects (Li et al., 20 Mar 2025).

A plausible implication is that the benchmark is designed not merely as a leaderboard instrument but as a stress test for compositional motion control under increasing object cardinality. That interpretation is consistent with the benchmark’s explicit object-number stratification and its support for dense, moderate-sparsity, and high-sparsity trajectory conditions.

4. MagicBench in fine-grained text-to-image artifact assessment

Within MagicMirror, MagicBench is an automated benchmark for fine-grained assessment of physical artifacts in text-to-image generation. Its purpose is to move beyond coarse, preference-based, or aesthetic-focused benchmarks by evaluating generated images for concrete physical artifacts such as anatomical errors, object distortions, and illogical interactions (Wang et al., 12 Sep 2025).

The benchmark is built from 800 carefully designed prompts. These prompts are systematically stratified into categories covering single, double, and multiple humans; single and multiple animals; and single, multiple, and complex objects, with 100 prompts in each category. Prompt diversity spans subjects, scenes, photographic styles, and angles, and final instructions are appended to ensure that the main subjects are clearly present (Wang et al., 12 Sep 2025).

Benchmark construction proceeds by having each candidate text-to-image model generate one image per prompt, yielding 800 images per model. A separate general-purpose VLM then verifies whether the required subject is actually present, because some models may omit hard subjects. Only images passing this verification stage are forwarded to MagicAssessor for artifact identification and fine-grained label assignment (Wang et al., 12 Sep 2025).

MagicBench in this setting measures the prevalence and types of physical artifacts, focusing on human, animal, and object anatomy and morphology, irrational element interactions, attributes such as color, material, and proportions, and aggregate artifact rates summarized as an overall normality score. The benchmark therefore answers not only whether an image is flawed, but also which kind of flaw is present and how frequently it appears by category or context (Wang et al., 12 Sep 2025).

5. Taxonomy, MagicAssessor, and scoring in the MagicMirror benchmark

The text-to-image version of MagicBench relies on the taxonomy established by MagicData340K and operationalized by MagicAssessor. The hierarchy has L1, L2, and L3 levels. L1 is a binary distinction between Normal and Artifact. L2 contains five main artifact types: Abnormal Human Anatomy, Abnormal Animal Anatomy, Abnormal Object Morphology, Irrational Element Interaction, and Irrational Element Attributes. L3 provides category-specific subtypes such as hand structure deformity, abnormal posture, and spatial overlap. A single image may receive multiple artifact labels (Wang et al., 12 Sep 2025).

MagicAssessor is the model backbone of this benchmark. It is a specialized vision-LLM built on Qwen2.5-VL-7B and fine-tuned for fine-grained artifact detection and reasoning. Training is described as a two-stage process: Supervised Fine-Tuning on a Chain-of-Thought subset, followed by Group Relative Policy Optimization (GRPO) for sequence-level optimization under custom reward design. The model outputs a structured reasoning trace and a dictionary-like output indicating which artifacts, if any, are present (Wang et al., 12 Sep 2025).

To address class imbalance and reward hacking, the paper specifies a multi-bucket sampling strategy with five buckets—Normal and four main L2 artifact labels—sampled at a 4:1:1:1:1 ratio. It also uses challenging positive upsampling so that difficult positive examples, such as correctly rendered hands, appear more often and the model does not collapse into overly pessimistic heuristics (Wang et al., 12 Sep 2025).

The reward system decomposes into format reward $\downarrow$0, hierarchical rewards $\downarrow$1, $\downarrow$2, $\downarrow$3, and a consistency reward $\downarrow$4. The final reward is

$\downarrow$5

This weighting scheme prioritizes correct format and high-level label accuracy while also enforcing consistency between reasoning and structured predictions (Wang et al., 12 Sep 2025).

Scoring in MagicBench is then computed per label or category from the number of images assigned a particular artifact label relative to the total number of relevant images. The Overall Score is defined by the proportion of images labeled as Normal. Reported benchmark findings include that no model is artifact-free, that unified models generally outperform diffusion-only models, that GPT-image-1 performs best overall, that FLUX.1-dev leads among diffusion models, and that human anatomy remains the most challenging type for all models (Wang et al., 12 Sep 2025).

6. Comparative interpretation and recurrent design themes

The two benchmarks share a family resemblance at the level of evaluation philosophy, even though they target different tasks. Both reject a single undifferentiated quality score in favor of fine-grained diagnostics: the video benchmark stratifies by number of moving objects and measures both appearance and control; the text-to-image benchmark stratifies by artifact type and separates subject verification from artifact assessment (Li et al., 20 Mar 2025, Wang et al., 12 Sep 2025).

They also differ sharply in what they treat as the principal failure mode. In the video benchmark, the central issue is whether a model can preserve object consistency and adhere to dense-to-sparse trajectories as the number of controlled objects increases. In the text-to-image benchmark, the central issue is whether a model can avoid physical artifacts in anatomy, morphology, interaction, and attributes, even when the required subject is present (Li et al., 20 Mar 2025, Wang et al., 12 Sep 2025).

A common source of confusion is nomenclature rather than method. "MagicBench" does not denote a single benchmark with a stable protocol across papers; it denotes at least two separate benchmarks introduced in distinct subfields in 2025. A practical consequence is that citations to "MagicBench" require disambiguation by paper title, modality, or accompanying framework name. It is likewise distinct from MAGIC, the benchmark for inter-context knowledge conflicts in retrieval-augmented generation (Lee et al., 29 Jul 2025).

This suggests that "MagicBench" functions less as a singular canonical benchmark than as a reused label for benchmark construction within different "Magic"-named research programs. For scholarly use, the technically precise referents are therefore MagicBench in MagicMotion for trajectory-controllable video generation and MagicBench in MagicMirror for fine-grained text-to-image artifact assessment.