RefVIE-Bench: Video Editing Benchmark

• RefVIE-Bench is a curated evaluation benchmark for video editing that integrates textual instructions and visual reference images without paired target videos.
• It assesses models on dual task families—Subject Reference and Background Replacement—by focusing on identity, temporal consistency, and physical integration.
• The benchmark employs a state-of-the-art multimodal language model judge with a hierarchical scoring system, ensuring fine-grained evaluation of edited video outputs.

Searching arXiv for the cited paper and any closely related benchmark/dataset context. RefVIE-Bench is a curated evaluation benchmark for instruction-reference-following in video editing, introduced together with the RefVIE dataset and the Kiwi-Edit framework in "Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance" (Lin et al., 2 Mar 2026). It is designed to assess whether a model can simultaneously comply with a textual instruction, adhere to a visual reference image, and preserve temporal coherence and realistic physical integration in the edited video. Unlike RefVIE, which is a large-scale training resource of synthesized quadruplets, RefVIE-Bench is a manually verified test set built for evaluation rather than training, and it operates without paired ground-truth edited videos (Lin et al., 2 Mar 2026).

1. Definition and task formulation

RefVIE-Bench operationalizes the evaluation of "instruction-reference-following" through benchmark items structured as triplets $$(V_{\mathrm{src}}, I_{\mathrm{ref}}, T_{\mathrm{inst}})$$: an original source video, a reference image, and an instruction text (Lin et al., 2 Mar 2026). A model is required to generate an edited video conditioned on this triplet, after which the output is judged for reference adherence, instruction compliance, and temporal and physical realism. The absence of paired target videos is a defining property of the benchmark; evaluation is performed by an automated judge rather than by direct comparison to a ground-truth edit.

The benchmark is organized around two task families. In Subject Reference, evaluation concerns whether the edited subject matches the visual identity of $I_{\mathrm{ref}}$, follows $T_{\mathrm{inst}}$, and remains temporally and physically consistent. In Background Replacement, evaluation concerns whether the new background matches $I_{\mathrm{ref}}$, preserves the foreground from $V_{\mathrm{src}}$, and composes harmoniously with it (Lin et al., 2 Mar 2026).

This formulation distinguishes RefVIE-Bench from instruction-only video editing benchmarks. Natural language alone is treated as insufficient for precise visual specification, especially for identity and fine-grained appearance, so the benchmark explicitly tests joint conditioning on text and reference image. A plausible implication is that RefVIE-Bench targets a harder control regime than conventional instruction-following evaluation, because success depends on multimodal fidelity rather than semantic plausibility alone.

2. Composition, scope, and curation

RefVIE-Bench is described as manually verified and purpose-built for evaluation. The benchmark text reports two totals in different locations: the introduction states "RefVIE-Bench, a benchmark of 100 manually verified samples," whereas the benchmark section states "RefVIE-Bench, comprising 110 manually verified triplets $$(V_{\mathrm{src}}, I_{\mathrm{ref}}, T_{\mathrm{inst}})$$" (Lin et al., 2 Mar 2026). The detailed breakdown is 70 Subject Reference samples and 40 Background Replacement samples, which sums to 110 and therefore agrees with the benchmark section.

The Subject Reference portion emphasizes object insertion and swap scenarios across diverse categories, including vehicles, animals, clothing, and furniture, with particular attention to identity and fine-grained appearance. The Background Replacement portion focuses on replacing the environment while preserving foreground dynamics, with emphasis on matting and compositing quality (Lin et al., 2 Mar 2026). Bench allocation is heavier toward Subject Reference because that setting spans broader object diversity, whereas Background Replacement is treated as a more unified environment-change domain.

The benchmark is manually curated through a three-stage manual verification procedure to ensure quality and diversity (Lin et al., 2 Mar 2026). This curation protocol is explicitly distinct from the synthetic data pipeline used to construct the RefVIE training dataset. No benchmark-specific statistics are reported for video length, resolution, or instruction-length distributions. The paper does report duration statistics for the training dataset RefVIE, where most clips are 80-110 frames, but it does not provide the corresponding distributions for RefVIE-Bench (Lin et al., 2 Mar 2026).

This selective reporting matters for interpretation. Because the benchmark is manually verified and relatively compact, it is positioned as a high-quality evaluation set rather than a statistically exhaustive sample of all possible video editing conditions. This suggests that benchmark conclusions are strongest for controlled multimodal adherence and less suited to fine-grained stratified analysis by clip duration, resolution, or linguistic complexity.

3. Evaluation protocol and scoring rubric

RefVIE-Bench does not use a reference edited video for scoring. Instead, outputs are assessed by a state-of-the-art multimodal LLM, described in the text as Gemini 3 / Gemini 2.5, with appendix prompts labeled "Gemini" (Lin et al., 2 Mar 2026). The judge assigns scores on a 1-5 scale according to task-specific criteria.

For Subject Reference, the rubric contains three scores:

1. Identity Consistency: the degree to which the inserted or swapped object matches the reference image in identity and fine details.
2. Temporal Consistency: the stability of structure and texture across frames.
3. Physical Integration: tracking, occlusions, shadows, perspective, and lighting.

For Background Replacement, the rubric likewise contains three scores:

1. Reference Fidelity: semantic and stylistic match of the background to $I_{\mathrm{ref}}$ and preservation of the foreground from $V_{\mathrm{src}}$.
2. Matting Quality: edge stability, temporal stability, and handling of hair or transparency.
3. Visual Harmony: integration of perspective, scale, lighting, and color grading (Lin et al., 2 Mar 2026).

A hierarchical cap rule is imposed in both tasks. For Subject Reference, the second and third scores cannot exceed Identity Consistency; for Background Replacement, Matting Quality and Visual Harmony are capped by Reference Fidelity. The appendix prompts enforce this constraint explicitly (Lin et al., 2 Mar 2026). The purpose of the cap is logical consistency: poor reference fidelity should preclude high secondary scores.

The benchmark intentionally avoids CLIP Score and FID. The stated rationale is that CLIP captures high-level semantics only, while FID measures distribution-level statistics rather than per-sample detailed fidelity (Lin et al., 2 Mar 2026). No formal mathematical definitions are introduced for alternative metrics such as LPIPS, retrieval similarity, or cross-modal feature alignment within RefVIE-Bench; the paper explicitly opts for the MLLM-judge rubric instead.

This evaluation design addresses a common misconception that generic perceptual metrics are sufficient for reference-guided editing. In the RefVIE-Bench formulation, the central problem is not merely realism or semantic relevance, but fine-grained adherence to a supplied visual reference under temporal constraints. The reliance on a proprietary MLLM judge, however, also introduces calibration and bias concerns noted by the authors (Lin et al., 2 Mar 2026).

4. Relationship to RefVIE and the Kiwi-Edit framework

RefVIE-Bench is tightly coupled to the broader RefVIE ecosystem but serves a different role. RefVIE is a large-scale, open-source training resource of 477K quadruplets $$(V_{\mathrm{src}}, T_{\mathrm{inst}}, I_{\mathrm{ref}}, V_{\mathrm{tgt}})$$ generated from an initial 3.7M pool through a four-stage scalable pipeline (Lin et al., 2 Mar 2026). That training pipeline consists of source aggregation and filtering, grounding and segmentation, reference image synthesis, and quality control with post-processing. Data sources include Ditto-1M, ReCo-500K, and OpenVE-3M; filtering uses EditScore, with thresholds of $\geq 6$ for instruction tuning and $I_{\mathrm{ref}}$0 for reference-guided data, restricted to Local Modification or Background Replacement tasks (Lin et al., 2 Mar 2026).

Grounding and segmentation are performed with Qwen3-VL-32B for coarse boxes on the first frame of the target video and SAM3 for pixel-level masks. Reference image synthesis uses Qwen-Image-Edit-2511: for background tasks, the foreground is removed and the target frame is inpainted to create a clean background reference image; for local edits, the edited object is extracted, placed on a clean background, and tightly cropped to produce a high-fidelity subject reference (Lin et al., 2 Mar 2026). The final yield is 477K quadruplets after MLLM verification, CLIP feature-based de-duplication, and removal of problematic aspect ratios or resolutions.

RefVIE-Bench differs from this pipeline in two key respects. First, it is manually verified rather than synthetically generated. Second, it contains triplets rather than quadruplets and therefore does not supply a target video for direct supervision or direct metric comparison (Lin et al., 2 Mar 2026). Its function is to test whether models trained on resources such as RefVIE can generalize to controlled multimodal editing scenarios.

The benchmark is introduced together with Kiwi-Edit, whose architecture and curriculum are explicitly linked to benchmark performance. Kiwi-Edit uses Qwen2.5-VL-3B with LoRA as an MLLM encoder for interleaved inputs $I_{\mathrm{ref}}$1, with two conditioning pathways: learnable instructional queries via a Query Connector and dense visual tokens from $I_{\mathrm{ref}}$2 via a Latent Connector (Lin et al., 2 Mar 2026). The DiT backbone is Wan-TI2V-5B. Source video features are injected element-wise into the noisy latent with a learnable timestep-dependent scalar:

$I_{\mathrm{ref}}$3

Reference image features are patch-embedded and concatenated to the input sequence. Training uses a Flow Matching objective:

$I_{\mathrm{ref}}$4

The training curriculum has three stages: alignment on image editing triplets, instructional tuning on mixed image and video instruction data, and reference-guided fine-tuning using RefVIE quadruplets with a mixture ratio of image instruction : video instruction : video+reference = 2:1:1 (Lin et al., 2 Mar 2026). The paper maps these design choices directly onto RefVIE-Bench dimensions: learnable queries are associated with stronger instruction adherence, the Latent Connector with stronger reference fidelity, element-wise source injection with $I_{\mathrm{ref}}$5 with better temporal consistency and physical integration, and progressive curriculum with more stable optimization (Lin et al., 2 Mar 2026).

5. Reported results and empirical interpretation

RefVIE-Bench results are reported for Runway Aleph, Kling-O1, and two Kiwi-Edit training regimens: Ours (All data) and Ours (Ref. data only) (Lin et al., 2 Mar 2026). Scores are on the 1-5 scale defined by the MLLM judge.

For Runway Aleph, Subject Reference scores are Identity 3.79, Temporal 3.65, and Physical 3.58; Background Replacement scores are Reference 3.33, Matting 2.81, and Video Quality 2.58; Overall is 3.29 (Lin et al., 2 Mar 2026).

For Kling-O1, Subject Reference scores are Identity 4.75, Temporal 4.66, and Physical 4.60; Background Replacement scores are Reference 3.95, Matting 3.21, and Video Quality 2.75; Overall is 3.99 (Lin et al., 2 Mar 2026).

For Ours (All data), Subject Reference scores are Identity 3.51, Temporal 2.96, and Physical 2.91; Background Replacement scores are Reference 3.40, Matting 2.58, and Video Quality 2.40; Overall is 2.96 (Lin et al., 2 Mar 2026).

For Ours (Ref. data only), Subject Reference scores are Identity 3.98, Temporal 3.40, and Physical 3.34; Background Replacement scores are Reference 3.72, Matting 2.90, and Video Quality 2.51; Overall is 3.31 (Lin et al., 2 Mar 2026).

Two observations are stated explicitly. First, Kiwi-Edit trained only on RefVIE reference data reaches Overall 3.31, slightly surpassing Runway Aleph's 3.29, while showing competitive Subject Identity at 3.98 and Background Reference Fidelity at 3.72 (Lin et al., 2 Mar 2026). Second, Kling-O1 leads in absolute scores, which the authors attribute likely to scale and private training corpus.

Ablations reported as relevant to benchmark behavior further connect architecture to evaluation outcomes. In the reference condition design ablation, Queries only obtains Score@Subject = 3.20, whereas Queries + Reference Latent obtains Score@Subject = 3.30, indicating that explicit injection of dense visual latents improves subject-reference fidelity (Lin et al., 2 Mar 2026). Additional conditioning and curriculum ablations are reported on instruction tasks rather than directly on RefVIE-Bench, but the paper links them to structural preservation and fine-grained editing, which plausibly affect Temporal Consistency and Physical Integration in the benchmark setting.

The results also highlight a discrepancy between open and closed systems. The benchmark shows that an open-source model trained specifically on curated reference data can become competitive with a commercial baseline in overall score, yet the highest absolute performance remains with a closed-source system. This suggests that RefVIE-Bench measures not only algorithmic conditioning quality but also the downstream effects of model scale and corpus breadth.

6. Qualitative behavior, reproducibility, and limitations

Qualitative examples in the paper illustrate accurate localization and adherence to reference identity, including cases such as hat placement and preserved subject identity under background style changes; red bounding boxes in one figure highlight maintained subject consistency despite drastic background changes (Lin et al., 2 Mar 2026). The paper also notes failure modes consistent with the scoring rubric, including temporal flicker, edge halos during matting, lighting or perspective mismatch, and partial identity drift when reference guidance is weak.

A specific challenge identified by the authors is degradation in background performance after Stage 3 training, attributed to training data that is "heavily biased toward local changes" (Lin et al., 2 Mar 2026). This bias provides a direct explanation for weaker Background Replacement behavior relative to Subject Reference in some settings. A plausible implication is that benchmark outcomes are sensitive not only to model architecture but also to the task balance of the fine-tuning distribution.

RefVIE-Bench is released together with datasets, models, and code through the Kiwi-Edit repository and project page (Lin et al., 2 Mar 2026). The appendix provides the exact Gemini evaluation prompts for Subject Reference and Background Reference editing tasks, including input specification, output formatting, brief reasoning, and the cap constraints. A derived evaluation workflow consists of downloading benchmark triplets, generating edited videos with a candidate model, scoring outputs with the provided Gemini prompts, and aggregating per-sample scores into category means and an overall mean as in the reported table (Lin et al., 2 Mar 2026).

Several limitations are acknowledged. The benchmark is relatively small at 110 samples by the detailed count, even though it is manually verified for diversity. Automated judging depends on a proprietary MLLM, so bias and calibration remain concerns. The benchmark intentionally avoids CLIP, FID, and LPIPS because of their limitations for fine-grained reference adherence, but future work could complement MLLM judgment with more robust feature-space metrics for reference fidelity and temporal consistency (Lin et al., 2 Mar 2026). Coverage is also limited by the lack of detailed benchmark metadata for resolution and frame-length distributions, and by its focus on image references rather than reference videos, multi-reference inputs, or 3D constraints.

In sum, RefVIE-Bench provides a human-verified test set of 110 triplets, a task-specific MLLM scoring rubric with hierarchical consistency constraints, released prompts for reproducible evaluation, and comparative measurements against both commercial systems and Kiwi-Edit variants (Lin et al., 2 Mar 2026). Within the Kiwi-Edit study, it functions as the principal benchmark for assessing whether a video editing system can jointly satisfy textual instructions and image-based reference control under temporal and physical realism constraints.