YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal

Abstract: Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26.

• The paper introduces a novel selective token processing framework (YOSE) that accelerates DiT-based VOR by reducing redundant computations.
• It employs Batch Variable-length Indexing and a Diffusion Process Simulator to dynamically process only essential tokens while preserving context.
• The approach achieves up to 3.3x acceleration with improved PSNR and SSIM, demonstrating efficient video object removal on benchmarks like YouTube-VOS.

Efficient DiT-based Video Object Removal via Selective Token Processing: An Analysis of the YOSE Framework

Introduction

The increasing computational demands of Diffusion Transformer (DiT)-based frameworks for video object removal (VOR) have motivated advances in model efficiency and scalability. Although models such as MiniMax Remover and VACE achieve strong spatial and temporal fidelity, their dense computation across the full spatiotemporal token space results in significant inference latency, particularly when the area to be synthesized is a small masked region. "YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal" (2604.27322) addresses this inefficiency by introducing a mask-aware, token-level acceleration framework that preserves the core advantages of DiT-based VOR while reducing computational redundancy.

Core Contributions

YOSE is a fine-tuning framework designed to be minimally invasive and compatible with existing DiT-based VOR architectures. The principal contributions are:

1. Batch Variable-length Indexing (BVI): A differentiable and dynamic token selection mechanism that operates on the masked regions, enabling variable-length token processing within and across batch samples.
2. Diffusion Process Simulator (DiffSim): A module that simulates the semantic influence of unmasked (outer) regions during generative self-attention, thus maintaining boundary consistency and global coherence without unnecessary token computation.
3. Boundary Fusion Strategy: An overlap-aware normalization approach that aligns local mean-variance statistics at mask boundaries, significantly mitigating artifacts at the reconstructed-original interface.

This design achieves inference time scaling linearly with the mask ratio, in explicit contrast to the constant (and thus redundant) complexity of prior approaches, while maintaining perceptual and quantitative fidelity to baseline solutions.

Methodology

Batch Variable-length Indexing (BVI)

YOSE replaces standard hard-masking tensor operations—which disrupt gradient flow and batch parallelization—with a differentiable grid-sample mechanism. Forward and backward indices are computed for each sample to select and re-map masked tokens, supporting variable numbers of selected tokens in each batch element. By padding to the batch maximum, efficient batch-wise computation is preserved. Thus, the computational complexity of YOSE is $O(yN)$, where $y$ is the mask ratio and $N$ is the total number of tokens, yielding substantial acceleration for low-mask-ratio VOR scenarios.

Diffusion Process Simulator (DiffSim)

To avoid loss of contextual dependencies between masked (inner) and unmasked (outer) regions, DiffSim uses learnable parameters to blend the predicted mask latent and the noise residual from the unmasked region. These proxy key-value (KV) features are introduced into each DiT block via attention, enabling inner tokens to attend to a contextually simulated outer region without processing every token. Importantly, only three small parameter groups are trained (combining parameter $G$, scaling $S$, and bias $B$), yielding an efficient adaptation to existing models.

Fusion and Boundary Alignment

Residual discrepancies at the boundaries of the reconstructed region are addressed by mean-variance alignment across a dilated mask overlap. The strategy enforces statistical matching between the synthesized and preserved video content, enabling smooth visual transitions and suppressing visible seams. The masked area reconstruction is finally blended with the original sequence via a weighted mask.

Experimental Results

Computational Efficiency

On standard benchmarks (YouTube-VOS and DAVIS), YOSE demonstrates up to a 2.5x–3.3x acceleration in typical low-mask-ratio scenarios (≤20%), supported by both theoretical FLOPs analysis and observed running time. The inference speed advantage rapidly diminishes only as the mask ratio approaches full-frame, where the computational burden converges with that of the baseline model.

Quantitative and Qualitative Performance

YOSE-integrated MiniMax Remover and VACE models consistently match or exceed the original in all core VBench metrics (Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality). For MiniMax Remover, YOSE preserves background PSNR (31.01 dB vs. 30.33) and maintains negligible deviation in perceptual metrics (all differences within $1 \times 10^{-2}$). The improvement is more pronounced for VACE, where YOSE yields a background PSNR increase of 5.47 dB and SSIM increase of 0.07 on YouTube-VOS, indicating better preservation of unmasked regions due to the mask-causal token selection and contextual simulation.

Qualitative comparisons further validate that YOSE results in more coherent edge continuity and fewer boundary artifacts, outperforming baselines especially in preserving background realism and preventing over-modification outside the masked area.

Component Analysis

Ablation studies confirm that both Lat Nis and Latmask are critical for effective context simulation in DiffSim. Exclusion of the fusion strategy introduces edge artifacts and minor but measurable degradations in VBench and background metrics, demonstrating the necessity of each component for optimal synthesis fidelity.

Implications and Future Directions

By realizing efficient, mask-aware selective token processing, YOSE signals a shift from uniform full-frame diffusion-based VOR toward content-adaptive computation. This enables real-time or near-real-time VOR on high-resolution video, making high-fidelity object removal feasible for interactive and production post-processing pipelines. The plug-in nature of YOSE implies immediate applicability to other token-based diffusion models beyond the current VOR context, including general video inpainting and localized video editing.

From a theoretical standpoint, YOSE also motivates further study into differentiable, dynamic attention indexing, sparsity-aware diffusion process modeling, and the seamless blending of generated and preserved content at high granularity. Adapting similar mechanisms to Transformer-based video generation, editing, and even multimodal synthesis tasks is a plausible avenue for future research.

Conclusion

YOSE provides an effective framework for efficient, selective token processing in DiT-based video object removal. Through BVI, DiffSim, and boundary-aware fusion, it achieves linear scaling of inference with mask size, maintains or improves baseline reconstruction fidelity, and is compatible with extant DiT-based architectures. The approach offers a practical and broadly applicable solution for real-time, high-quality, and resource-efficient video editing applications, setting a foundation for further research in mask-aware token-efficient generative modeling.