- The paper introduces PROVE, a unified evaluation framework that proposes RC-S and RC-T metrics to assess spatial and temporal coherence in object removal.
- It employs a sliding-window MMD approach using DINOv2 features to detect localized spatial and temporal artifacts with high sensitivity.
- PROVE-Bench, a dual-tier benchmark suite, validates the framework on both synthetic and real-world datasets, demonstrating superior correlation with human perceptual judgments.
Motivation and Problem Statement
The task of object removal in visual media—images and videos—poses significant evaluation challenges due to its fundamentally ill-posed, one-to-many nature. Multiple plausible restorations exist for any erased region, precluding the definition of a singular ground truth. Conventional evaluation metrics are misaligned with human perception: full-reference (FR) metrics (e.g., PSNR, SSIM, LPIPS) reward conservative, copy-paste behaviors rather than authentic contextual restoration, while no-reference (NR) metrics (e.g., ReMOVE, CFD) are systematically biased towards blur and lack sensitivity to complex occlusion or hallucination artifacts. Global temporal metrics (e.g., TC, TF) dilute artifact detection by aggregating features over the entire frame rather than focusing on edited regions.
RC Metrics: Removal Coherence Spatial and Temporal
To address these limitations, the paper proposes Removal Coherence (RC), an evaluation framework comprising RC-S (spatial) and RC-T (temporal) metrics. Both leverage a local distribution matching paradigm in deep feature space using DINOv2 representations.
RC-S operates via sliding-window Maximum Mean Discrepancy (MMD) comparisons between features extracted from masked (restored) and unmasked (background) regions. This finely-grained regional analysis enables detection of local spatial incoherence, outperforming similarity-based aggregation methods in sensitivity to artifacts and corruption, particularly blur-induced distortions.
RC-T extends this to the temporal domain by synchronously cropping adjacent frames with shared union masks, allowing comparison of feature distributions exclusively within intersected restored regions. This focused approach detects localized temporal instability (e.g., flickering, abrupt changes) that escapes global temporal metrics.
Empirical results demonstrate that RC metrics achieve substantially higher correlation with human rankings across image and video object removal benchmarks. RC-S is robust against the "Blur is Clean" bias—producing monotonic quality degradation with increasing blur intensity, in contrast to existing metrics which erroneously favor blurred outputs.
PROVE-Bench: Comprehensive Benchmark Suite
The paper presents PROVE-Bench, a two-tier benchmark suite:
- PROVE-M: 80 videos captured in real-world settings, each consisting of input video, object mask, and ground-truth target-free video. Motion augmentation via geometric transformations (cropping, scaling, translation) simulates realistic user-captured dynamics, increasing evaluation difficulty and stress-testing both models and metrics.
- PROVE-H: 100 challenging real-world videos devoid of ground truth, targeting unconstrained scenarios: crowds, dynamic backgrounds, complex reflections, highly textured scenes, fast motion. Masks are automatically generated by SAM3 and left unrefined to preserve practical segmentation imperfections.
PROVE-Bench bridges the realism-evaluability gap inherent in prior datasets, uniquely combining real-world authenticity, paired references, and unconstrained challenge coverage.
Empirical Validation and Ablation
Extensive benchmarking includes synthetic benchmarks (ROSE-Bench), real-world datasets (DAVIS), and the proposed PROVE-Bench. State-of-the-art removal models are evaluated using both conventional and proposed metrics. Human studies (Kendall’s τ, Spearman correlation) and GPT-4o-based evaluations confirm the superior alignment of RC metrics with perceptual judgments.
Ablation studies reveal the importance of DINOv2 features (highest sensitivity to spectral distortions), sliding-window locality, and MMD as the discrepancy metric. Removal of any component degrades correlation with human rankings; locality is critical for exposing region-specific artifacts, and MMD is inherently more sensitive to distribution shifts than cosine similarity.
Temporal sensitivity analyses show RC-T alone responds monotonically to synthetic temporal corruptions (Random Drop, Random Replace, mask blur), while conventional TC/TF remain indifferent or produce counter-intuitive trends. This establishes RC-T as the only robust temporal metric for localization.
Analysis of Metric Biases
The paper rigorously dissects failure modes in traditional metrics:
- FR metrics: Exhibit copy-paste bias, reward background preservation, and penalize high-frequency details due to regression-to-the-mean effects. Mask-only variants are heavily dependent on GT quality and cannot generalize to no-GT settings.
- NR metrics: Suffer from "Blur is Clean" and "Original is Better" biases, incorrectly favoring blurry or unedited inputs, especially in complex occlusion scenarios. Heuristics in CFD are vulnerable to over-segmentation and misclassification via SAM, while ReMOVE's aggregation strategy dilutes localized differences.
- Global temporal metrics: Are insensitive to localized restoration errors, failing to penalize flickering or instability within the restored regions.
RC metrics and PROVE-Bench directly address these vulnerabilities through local distribution analysis and targeted benchmarking.
Practical and Theoretical Implications
The RC framework fundamentally reframes evaluation in object removal away from reference-based fidelity and global aggregation, aligning directly with perceptual qualities—spatial and temporal coherence. It enables rigorous diagnosis of both restoration failures and hallucination artifacts, even without ground truth. Practically, this will facilitate standardized benchmarking, model development, and deployment in unconstrained real-world scenarios. Theoretically, it offers an exemplar for designing task-aligned metrics leveraging semantic feature space and kernel-based distributional comparison.
Future directions highlighted include the expansion of benchmarks to simulate free-camera effects (3D changes, rolling shutter, motion blur), improved pairing methodologies (robotic arm-mounted capture), and additional coverage of complex physical side effects.
Conclusion
PROVE introduces a unified, perception-aligned evaluation framework for visual object removal, comprising RC metrics and a comprehensive benchmark suite. RC-S and RC-T outperform existing metrics in correlation with human and automated judgments, robustly diagnosing spatial and temporal failures. PROVE-Bench establishes a new standard for realism and evaluability, facilitating the next generation of object removal research in both images and videos.
