DeeptraceReward: AI Video Evaluation Framework

• DeeptraceReward is a comprehensive framework that defines and categorizes human-perceived deepfake traces in AI-generated videos.
• It provides a rigorously annotated dataset with spatial and temporal labels, enabling detailed performance metrics including IoU and explanation scores.
• The paradigm advances reward modeling by integrating multimodal, spatiotemporal, and linguistic feedback for improved model interpretability and precision.

DeeptraceReward is a research framework and dataset for evaluating and improving the trustworthiness and perceptual alignment of AI-generated videos via human-perceived “deepfake traces.” It also refers, more broadly, to an emerging research paradigm using fine-grained, human-anchored reward signals—termed generically "DeepTraceReward"—to oversee, diagnose, and steer generative models in complex environments, including but not limited to video and TableQA. The approach foregrounds stepwise, interpretable feedback at the artifact or reasoning-trace level, providing both a new class of benchmarks and reward modeling strategies for LLMs, multimodal models, and reasoning agents (Fu et al., 26 Sep 2025).

1. Dataset Foundation and Annotation Protocol

DeeptraceReward (capitalized to refer to the video benchmark) comprises 3,318 AI-generated and 3,318 matched real videos, annotated with 4,334 perceived “fake traces” by human experts. Each annotation includes:

• A natural-language explanation describing the artifact, for example, “The lamp’s metal post jitters unnaturally.”
• A spatial bounding box ⟨x₀, y₀, x₁, y₁⟩ localizing the artifact within video frames.
• Precise temporal localization via onset and offset timestamps (ts, te).

Annotation was performed using LabelBox, with subsequent expert consensus review and language standardization by GPT-4. These annotations consolidate into nine dominant movement-centric artifact categories, covering 90% of observed deepfake traces in the dataset (Fu et al., 26 Sep 2025).

2. Taxonomy of Deepfake Traces

DeeptraceReward identifies the following nine principal categories of video artifacts prevalent in AI-generated content:

Category	Example Description
Object distortion	Warped textures on a running shoe
Motion blurring	Handprint smear during a fast swipe
Object merging	Two boats fused at the hull
Object splitting	Bouncing ball fragmenting midair
Unexpected disappearance	Car vanishing suddenly
Inconsistent trajectory	Thrown apple arcing non-parabolically
Flicker or pixelation	Intermittent pixel noise on a sign
Geometry implausibility	Floating clock hands
Lighting or shading anomaly	Shadow direction reversing

These trace types form the scaffold for both bounding box and explanation-based reward modeling, closely reflecting the cues human judges use to discern machine-generated content (Fu et al., 26 Sep 2025).

3. Evaluation Metrics and Protocol

DeeptraceReward enables fine-grained quantitative model evaluation using several orthogonal axes:

• Binary classification accuracy: $\mathrm{Acc} = \frac{TP + TN}{P + N}$
• F1 Score for “fake” detection: precision, recall, and $F_1$ based on detection counts.
• Spatial grounding: Intersection-over-Union (IoU) between predicted and ground-truth bounding boxes.
• Temporal localization: Temporal IoU for overlap between predicted $[s_p, e_p]$ and annotated $[s_{gt}, e_{gt}]$.
• Explanation score: Judged by GPT-4.1 as 0/0.5/1, averaged over prompts.
• Overall score: Aggregate over accuracy, explanation, IoU, and time distance (normalized error).

This multifaceted approach allows benchmarking not just global classification but also the model’s explanatory, spatial, and temporal reasoning capabilities (Fu et al., 26 Sep 2025).

4. Reward Model Architecture and Training

The core reward model is a VideoLLaMA 3 7B multimodal LLM, which processes video frames with text prompts and emits four outputs:

1. Binary real/fake prediction,
2. Bounding-box coordinates,
3. Predicted artifact start time $t_p$,
4. Natural-language explanation.

Training bases the total loss on a weighted sum:

$$\mathcal{L} = \lambda_{\mathrm{cls}} \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{bbox}} \mathcal{L}_{\mathrm{bbox}} + \lambda_{\mathrm{time}} \mathcal{L}_{\mathrm{time}} + \lambda_{\mathrm{exp}} \mathcal{L}_{\mathrm{exp}}$$

where losses are:

• Cross-entropy classification,
• Smooth L1 loss on box coordinates,
• $L_1$ error on predicted start time,
• Negative log-likelihood for the explanation.

The model is directly supervised on all annotation modalities, allowing fine-grained feedback for reinforcement learning (RL), video quality assessment, or model benchmarking (Fu et al., 26 Sep 2025).

5. Empirical Performance and Analysis

After supervised fine-tuning on DeeptraceReward, the VideoLLaMA 3 7B reward model achieves large gains over GPT-5 (zero-shot) across all major error axes. Key findings include:

Metric	GPT-5 (Zero-Shot)	7B Reward Model
Accuracy	90.7%	99.4%
Explanation Score	40.9	70.6
BBox IoU	10.4	32.6
Time Distance (↓)	100.0	21.9
Overall Score	35.5	70.2

Statistical significance was established with $p < 0.01$. The reward model displays a consistent difficulty gradient: binary classification is easiest, then explanation, followed by spatial localization (IoU < 36%), and finally temporal labeling (largest error). Off-the-shelf LLMs tend to overpredict bounding boxes and fail on temporal localization, highlighting the value of joint spatiotemporal-linguistic reward learning (Fu et al., 26 Sep 2025).

6. DeepTraceReward in Reinforcement Learning and Oversight

The DeeptraceReward paradigm generalizes to other domains, notably reasoning agents and TableQA (RE-TAB (Kwok et al., 30 Jan 2026)). Here, reward modeling uses interpretable, stepwise feedback (“trace reward”)—in effect, measuring effort and alignment at each reasoning or transformation step. This differs from scalar or purely outcome-based reward by offering a trajectory-level oversight mechanism:

• In TableQA, stepwise reward functions such as lexically normalized LCS between intermediate table states and the question ensure answers emerge from the most interpretable, verifiable chains, yielding 10–40 percentage point accuracy gains and up to 25% inference cost reduction without model re-tuning (Kwok et al., 30 Jan 2026).
• In reasoning and RL settings, DeeptraceReward can incorporate chain-of-thought (CoT) oversight, structure-aware metrics, and detect shortcutting or hacking behavior by cross-referencing model decisions with supervised or unsupervised trace signals (Wang et al., 1 Oct 2025).

Paradigm elements, such as TRACE (Truncated Reasoning AUC Evaluation), further extend DeeptraceReward oversight by measuring how much of a model’s generated trace is necessary to pass a verifier, robustly detecting “shortcut” or “cheating” behavior invisible to CoT monitors (Wang et al., 1 Oct 2025).

7. Broader Impact and Research Directions

DeeptraceReward and its associated methodologies foreground a shift from outcome-only evaluation to human-aligned, explainable, and fine-grained reward modeling. This opens new directions for:

• Artifact-level RL from human feedback in video and vision-LLMs,
• Multi-axes monitoring and detection of reward hacking or model cheating,
• Spatiotemporal and explanatory benchmarking for generative and reasoning systems,
• Plug-and-play, training-free wrappers for interpretable oversight (as in RE-TAB),
• Integrating uncertainty, trajectory diversity, and adaptive truncation into model reward signals.

DeeptraceReward establishes a rigorous scaffold for socially aware, interpretable, and robust AI system evaluation and training, emphasizing not only what predictions are made but how and why they arise from model computation (Fu et al., 26 Sep 2025, Kwok et al., 30 Jan 2026, Wang et al., 1 Oct 2025).