Stable Video Object Removal (SVOR)

• Stable Video Object Removal (SVOR) is the process of erasing objects from video while preserving spatial, temporal, and physical coherence.
• Techniques in SVOR range from optical flow warping and diffusion-based methods to counterfactual learning that update shadows, reflections, and other side effects.
• Recent research emphasizes robust performance under imperfect masks, efficiency improvements, and addressing challenges in physics-aware and semantic consistency.

Searching arXiv for papers on stable video object removal and closely related benchmarks/methods. Stable Video Object Removal (SVOR) denotes video object erasure that preserves spatial integrity and temporal stability, and in more recent formulations also preserves physical plausibility by updating or removing shadows, reflections, occlusions, illumination changes, and other object-induced effects rather than merely filling a masked hole. Across the literature, the term is used in several closely related senses: as a general objective of artifact-free, temporally coherent removal; as the name of a specific robust diffusion framework; and, earlier, as the name of a synthesized benchmark dataset. The unifying requirement is that the edited sequence remain visually and temporally coherent while non-target content is preserved as faithfully as possible (Lou et al., 17 Jan 2026, Hu et al., 10 Mar 2026, Li et al., 7 Apr 2026).

1. Terminology, scope, and historical usage

The meaning of “stable” in video object removal has expanded over time. In earlier work, stability primarily meant spatio-temporal consistency: no flicker, no frame-to-frame drift, and no obvious inconsistency in the inpainted region. VORNet framed this as producing videos “without flicker or frame-to-frame drift,” combining optical-flow warping, image inpainting, recurrent refinement, and a temporal adversarial loss on its Synthesized Video Object Removal (SVOR) dataset derived from YouTube-VOS (Chang et al., 2019). AutoRemover, developed for autonomous driving videos, likewise emphasized temporally consistent inpainting under large ego-motion and explicitly extended masks to include shadows, reflecting an early recognition that object removal and side-effect removal are coupled (Zhang et al., 2019).

More recent work uses “stable” in a broader sense. The 2026 SVOR framework defines the target as “shadow-free, flicker-free, and mask-defect-tolerant object removal under real-world imperfections,” explicitly addressing abrupt motion, missing or inaccurate masks, and real side effects such as shadows and reflections (Hu et al., 10 Mar 2026). Physics-aware evaluation further broadens the concept: PVIR defines stability as requiring that the target vanish consistently across frames and that the video remain physically plausible, so that shadows, reflections, occlusions, and illumination effects caused by the object are updated or removed in a way that respects light transport and scene dynamics (Li et al., 7 Apr 2026).

A recurring source of confusion is that not every relevant method explicitly uses the acronym “SVOR.” The stochastic-bridge formulation, VOID, and Object-WIPER all target the same core objective—temporally coherent, physically or semantically consistent object-and-effect removal—but describe it through bridge modeling, counterfactual deletion, or training-free editing rather than through the SVOR label (Lou et al., 17 Jan 2026, Motamed et al., 2 Apr 2026, Kushwaha et al., 10 Jan 2026). This suggests that SVOR is best understood as a problem class rather than a single model family.

2. Task formulation and defining failure modes

A common formulation takes as input an original video and a target specification, usually a per-frame mask and optionally a textual instruction. PVIR standardizes the task as: given $(V, M, P)$, output the edited video $\hat{V}$ with the target and its physical derivatives removed while preserving non-target content (Li et al., 7 Apr 2026). The 2026 SVOR framework uses input video $x=\{x_1,\ldots,x_T\}$ and imperfect guidance masks $M=\{M_1,\ldots,M_T\}$, where defects may include erosion, holes, jitter, frame dropout, or coarse boxes; the goal is to remove the specified object and side effects while restoring a temporally consistent background (Hu et al., 10 Mar 2026).

Within this formulation, stability has several separable components. One is temporal consistency: the output should avoid flicker, ghosting, texture popping, and shape wobble across frames. Another is spatial or structural fidelity: the filled region should align with surrounding geometry and preserve non-target regions. A third is semantic and physical correctness: the system should remove not only the target object but also its causally linked manifestations, such as shadows, reflections, translucencies, emitted light, ripples, wakes, or changes in downstream object motion (Li et al., 7 Apr 2026, Chen et al., 28 May 2026, Kushwaha et al., 10 Jan 2026).

The literature identifies several recurrent failure modes. Physics-aware evaluation reports lingering shadows or reflections, “ghosting” silhouettes, blurring within and near the inpainted region, halos leaking beyond mask boundaries, temporal flicker, view-dependent specular mis-handling, semantic hallucinations in the filled region, and background inconsistency such as texture smearing or lighting drift in untouched areas (Li et al., 7 Apr 2026). The robust SVOR framework further emphasizes missed removals under abrupt motion, mask collapse under temporal compression, and shadow artifacts caused by synthetic-only training and domain shift (Hu et al., 10 Mar 2026). VOID adds a distinct category of failure: appearance-only inpainting can leave the scene dynamically implausible when the removed object had collisions, support relations, or manipulation effects that should alter downstream motion (Motamed et al., 2 Apr 2026).

A common misconception is that video object removal is merely video inpainting restricted to a masked region. Current physics-aware and counterfactual formulations reject that view. When a person is removed, the edit may also need to erase a cast shadow; when a car is removed, its shop-window reflection may need to disappear; when a supporting object is removed, another object may need to fall rather than remain suspended. In that sense, SVOR increasingly operates at the boundary between inpainting, scene understanding, and causal video editing (Li et al., 7 Apr 2026, Motamed et al., 2 Apr 2026).

3. Technical mechanisms for achieving stability

Earlier learning-based systems achieved stability primarily through geometric reuse of observed background. VORNet computes background-only optical flow, warps previously generated frames to the current time step, generates an additional image-inpainting candidate, selects the candidate most consistent with the previous output by LPIPS, and refines the result with a UNet-style encoder-decoder containing a ConvLSTM bottleneck and a temporal discriminator (Chang et al., 2019). AutoRemover extends this line by detecting shadows with a U-Net, using depth and camera pose to derive inter-frame flow $U^{m \to i}$, and combining geometry-guided warping with a memory-efficient multi-frames-to-one 3D feature assembler that aggregates contextual-attention and global guidance features across aligned frames (Zhang et al., 2019). These systems define stability largely through explicit alignment and multi-frame feature reuse.

Diffusion and DiT-based methods reframe stability as a property of latent generative trajectories conditioned on masks, context, and sometimes auxiliary understanding modules. The robust SVOR framework introduces Mask Union for Stable Erasure (MUSE), which preserves all target locations within each temporal compression window by

$$M_{\text{down}}[k] = \bigcup_{\tau \in W_k} M[\tau],$$

thereby preventing under-erasure under abrupt motion. It pairs MUSE with a decoupled Denoising-Aware Segmentation (DA-Seg) side branch using Denoising-Aware AdaLN, and a curriculum two-stage training scheme: self-supervised pretraining on unpaired real background videos with random masks, followed by refinement on ROSE synthetic pairs with degraded masks and side-effect-weighted losses (Hu et al., 10 Mar 2026). This design makes stability depend not only on temporal modeling but also on robustness to imperfect mask guidance.

Other diffusion systems attempt to improve the semantics of removal rather than only the mechanics of denoising. “From Understanding to Erasing” introduces Object-Induced Relation Distillation (OIRD), which transfers object–effect relational knowledge from DINOv2 to a video diffusion denoiser, and Framewise Context Cross-Attention (FCCA), which injects per-frame unmasked background context from CLIP-vision into every denoising block. For long sequences it adds Keyframe-Guided Propagation (KGP), which stabilizes outputs beyond the 81-frame training length (Liu et al., 2 Apr 2026). GenEraser instead uses a Multi-Conditional Mixture-of-Experts (MC-MoE), Bipartite Text guidance that separately specifies what to erase and how the scene should look after erasure, Learnable Deep CFG Fusion (LD-CFG), and a Decoupled Expert Architecture with a high-noise Locator and low-noise Preserver to mitigate the trade-off between semantic generalization and precise pixel-level preservation (Chen et al., 28 May 2026).

A distinct line of work replaces noise-initialized diffusion with source-anchored transport. “Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation” models removal as a VP-SDE bridge from the source latent to the target latent, so generation remains anchored to the input video rather than starting from Gaussian noise. In compact form,

$dX_t = f_\theta(X_t, I, M, t)\,dt + g(t)\,dW_t,$

with $I$ providing a structural prior and $M$ indicating the masked region. The same work introduces Adaptive Mask Modulation,

$h' = h \odot \bigl(M + (1-M)\cdot(1+\gamma)\bigr),$

which amplifies reliable background context while leaving masked embeddings unchanged, balancing fidelity and flexibility for large removals (Lou et al., 17 Jan 2026). This directly targets the SVOR requirement that non-masked areas remain stable while masked areas admit sufficient generative freedom.

Training-free approaches achieve stability through careful editing schedules inside pretrained text-to-video transformers. Object-WIPER localizes associated effects by combining text–visual cross-attention and visual self-attention, fuses them with the user mask, inverts the video to structured noise, reinitializes only foreground tokens, scales attentions during inversion and early denoising, and copies background value features saved during inversion back into later denoising steps. Its design is explicitly intended to preserve background priors while removing both the object and its associated visual effects (Kushwaha et al., 10 Jan 2026).

Efficiency-oriented work addresses a different but practically important dimension of stability: maintaining quality while reducing latency. YOSE introduces Batch Variable-length Indexing (BVI) to process only essential masked tokens and DiffSim to simulate the influence of unmasked tokens in DiT self-attention, so that inference time scales approximately linearly with the masked region rather than remaining constant with respect to mask size. At 480p, a 20% mask ratio yields approximately $\hat{V}$0 acceleration and about 25 FPS for MiniMax+YOSE, versus approximately 9.5 FPS for the baseline (Wu et al., 30 Apr 2026). MiniMax-Remover reduces complexity even further by removing textual input and cross-attention layers, training a two-stage remover with minimax “bad noise” distillation, and achieving state-of-the-art results with as few as 6 sampling steps and no CFG (Zi et al., 30 May 2025).

4. Physics-aware and counterfactual extensions

Physics-aware SVOR treats object removal as a physically grounded edit rather than a purely visual repair. PVIR is explicit on this point: benchmark videos include cast shadows, mirror appearances and specular reflections, complex light transport, local illumination cues, indirect occlusion patterns, and motion-induced effects such as wakes and ripples. The Hard subset is specifically designed around stronger physics interactions including mirror reflections, specular highlights, and pronounced scene coupling, and typical hard prompts explicitly require removing both the object and its secondary manifestations (Li et al., 7 Apr 2026). In this setting, temporal stability is inseparable from physically consistent updating of light transport and geometry.

Several methods operationalize this broader notion of completeness. The robust SVOR framework adds side-effect-weighted diffusion losses that emphasize shadow and reflection regions, with $\hat{V}$1 when a side-effect map indicates a shadow or reflection and $\hat{V}$2 otherwise, using $\hat{V}$3 and $\hat{V}$4 in all experiments (Hu et al., 10 Mar 2026). “From Understanding to Erasing” constructs a side-effect mask during training from residual differences between original and object-absent frames, then distills object–side-effect relations so that the denoiser learns to jointly remove objects and induced effects without requiring explicit side-effect masks at inference (Liu et al., 2 Apr 2026). GenEraser uses effect-aware text tokens to capture weakly correlated effects beyond the spatial mask, including smoke, reflections, light, ripples, shadows, and deformations (Chen et al., 28 May 2026). Object-WIPER similarly targets shadows, reflections, mirror duplicates, translucencies, and motion trails through attention-driven associated-effect localization (Kushwaha et al., 10 Jan 2026).

VOID pushes the causal interpretation further. Its argument is that existing methods may correct appearance-level artifacts such as shadows and reflections yet still fail when the removed object participated in collisions, support relations, or manipulation. To address this, VOID trains on paired counterfactual videos rendered with Kubric and HUMOTO, uses a VLM to identify affected regions, encodes object and affected regions with a quadmask, and optionally performs a second pass with flow-warped noise when large motion change is predicted (Motamed et al., 2 Apr 2026). The result is a formulation of removal as “counterfactual world-edit” rather than local hole filling.

This shift has methodological consequences. Physically coupled hard cases—specular and mirror reflections, indirect illumination and occlusion, and motion-induced fluid effects—are repeatedly identified as the hardest stable-removal scenarios because they require coherent updates across frames and across scene surfaces rather than only within the nominal mask (Li et al., 7 Apr 2026). A plausible implication is that future SVOR systems will increasingly borrow tools from inverse rendering, scene decomposition, geometry-aware modeling, and causal video reasoning rather than relying solely on 2D texture synthesis.

5. Benchmarks, metrics, and empirical status

The evaluation landscape has diversified in parallel with the broadened problem definition. VORNet’s original SVOR dataset provided paired synthesized videos with and without a pasted target object, enabling objective evaluation against ground truth (Chang et al., 2019). Subsequent work introduced task-specific benchmarks for more realistic conditions: ROSE-Bench, DAVIS, and RORD-50 for robust removal under imperfect masks (Hu et al., 10 Mar 2026); CAMERA-Bench and Scene-Bench for real-captured and in-the-wild side-effect-heavy scenarios (Liu et al., 2 Apr 2026); WIPER-Bench for real associated-effect removal (Kushwaha et al., 10 Jan 2026); and PVIR for physics-aware removal with decoupled human scoring (Li et al., 7 Apr 2026).

PVIR provides the clearest formalization of multidimensional SVOR evaluation. It contains 95 high-quality real videos, partitioned into 57 Simple and 38 Hard examples, with 45 sequences from Inter4k and 50 from DAVIS2016, instance-accurate masks, structured removal prompts, and standardized 720p evaluation (Li et al., 7 Apr 2026). Each video–model pair is judged by at least 2 independent raters in randomized order along three 1–4 ordinal dimensions: Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE). For video $\hat{V}$5, rater $\hat{V}$6, and dimension $\hat{V}$7,

$\hat{V}$8

and per-dimension scores are then averaged over videos. Reporting includes 95% confidence intervals and bootstrap significance testing with $\hat{V}$9; the superiority of PISCO-Removal and UniVideo is statistically significant with $x=\{x_1,\ldots,x_T\}$0, especially on the Hard subset (Li et al., 7 Apr 2026).

The empirical picture from PVIR is that current systems are capable but far from complete. Overall on the 95 videos, CoCoCo scores IF 1.60, RQ 1.84, EE 3.07, Overall 2.17; UniVideo 3.06, 3.45, 3.53, Overall 3.35; DiffuEraser 2.89, 2.63, 3.52, Overall 3.01; and PISCO-Removal 3.62, 3.28, 3.58, Overall 3.49 (Li et al., 7 Apr 2026). The persistent drop on the Hard subset shows that physically coupled effects remain a major bottleneck. PVIR also reports that IF and RQ are moderately correlated with $x=\{x_1,\ldots,x_T\}$1, while RQ and EE are only weakly correlated in lower-performing methods, supporting the claim that visually pleasing fills can coexist with unintended background edits and that decoupled evaluation is necessary (Li et al., 7 Apr 2026).

Other benchmarks illuminate complementary aspects of SVOR. The robust SVOR framework reports on DAVIS, ROSE Bench, and RORD-50. On DAVIS it achieves ReMOVE 0.8800, GPT 12.34, mSSIM 0.9092, and TF 0.9510; on ROSE Bench it achieves PSNR 31.47, SSIM 0.9335, ReMOVE 0.9082, and GPT 13.18; and on RORD-50 it achieves PSNR 31.26, SSIM 0.9378, ReMOVE 0.9179, and GPT 13.82 (Hu et al., 10 Mar 2026). “From Understanding to Erasing” reports PSNR 32.2890, SSIM 0.9413, and LPIPS 0.1091 on ROSE-Bench, and PSNR 28.0855, SSIM 0.9448, and LPIPS 0.0726 on CAMERA-Bench, together with the best reported LLM ratings on Scene-Bench and the highest Average Human Ranking in its user study (Liu et al., 2 Apr 2026).

Automatic metrics remain less mature than human protocols. Object-WIPER proposes TokSim, which combines temporal consistency of edited foreground tokens across consecutive frames, dissimilarity between edited and original foreground tokens, and blending with nearby background tokens. On DAVIS it reports TokSim 32.80, BG-PSNR 23.02 dB, and FG-Flicker 16.37; on WIPER-Bench it reports TokSim 33.09, BG-PSNR 27.53 dB, and FG-Flicker 3.02, outperforming both training-based and training-free baselines on that metric (Kushwaha et al., 10 Jan 2026). This reflects a broader consensus that global image or video quality metrics alone are insufficient for SVOR because they may remain high even when the target object or its side effects persist.

Efficiency measurements have also become part of evaluation. YOSE reports up to $x=\{x_1,\ldots,x_T\}$2 speedup in 70% of cases while maintaining quality comparable to the baseline, with the overall FLOPs and latency growing approximately linearly with mask ratio (Wu et al., 30 Apr 2026). MiniMax-Remover reports 6-step, CFG-free inference with latency 0.18 s, GPU memory 8.2 GB, and 1.05B parameters for 33-frame 360p clips on A800 under the paper’s setup, and approximately 24 seconds per 81-frame 480p video on RTX 4090 (Zi et al., 30 May 2025). These results indicate that speed has become a first-class constraint for practical SVOR systems rather than a secondary implementation issue.

6. Limitations, open questions, and likely research directions

Despite rapid progress, the literature is consistent about several unresolved problems. Physics-aware work notes that a 95-video benchmark is strong for high-quality human evaluation but limited for training large models, that physics-aware prompts and mask refinement are labor-intensive, and that the field still lacks automatic metrics capable of detecting residual shadows or reflections, illumination drift, and fluid inconsistencies (Li et al., 7 Apr 2026). The robust SVOR framework identifies extreme mask sparsity, single-frame-mask regimes, and complex lighting or highly specular reflections as remaining failure cases, and notes that long videos may require chunked inference that reduces temporal context (Hu et al., 10 Mar 2026).

Causal extensions introduce a different set of limitations. VOID does not use an explicit physics engine at inference time; its physical plausibility emerges from paired counterfactual supervision and VLM guidance, so unusual camera angles, longer horizons, and higher resolutions remain challenging (Motamed et al., 2 Apr 2026). Training-free editing avoids finetuning costs but remains bounded by the reconstruction quality of the base video VAE and the inversion procedure, so background fidelity can still lag in hard cases, especially under very complex reflections, unseen backgrounds, or extreme fast motion (Kushwaha et al., 10 Jan 2026). Efficient DiT pruning and token selection help latency, but when the mask approaches full-frame coverage, acceleration diminishes and runtime converges toward the baseline (Wu et al., 30 Apr 2026).

Several papers converge on similar future directions. PVIR explicitly argues for differentiable rendering, light transport estimation, scene decomposition into geometry, albedo, and lighting, temporal radiance field priors, and other physically guided priors (Li et al., 7 Apr 2026). The robust SVOR framework points to fewer sampling steps or distillation for lower latency, and expanded side-effect supervision for hard lighting and reflection cases (Hu et al., 10 Mar 2026). UnderEraser shows that external relational understanding and internal background-grounded attention are complementary, suggesting that stronger object–effect reasoning modules may remain important even as base video generative models scale (Liu et al., 2 Apr 2026). GenEraser indicates that out-of-domain generalization depends on explicitly balancing semantic guidance and pixel preservation rather than optimizing a single monolithic remover (Chen et al., 28 May 2026).

Taken together, these results indicate that the central bottleneck in SVOR is no longer only temporal flicker. The harder problem is physical and causal incoherence: failure to update object-induced effects, failure to preserve untouched regions under strong generative priors, and failure to rewrite downstream dynamics when the removed object was an active participant in the scene. In that sense, the field is moving beyond 2D texture filling toward physically grounded, semantically complete, and computationally practical video object removal (Li et al., 7 Apr 2026).