IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning (2512.15635v1)

Abstract: We propose \textbf{IC-Effect}, an instruction-guided, DiT-based framework for few-shot video VFX editing that synthesizes complex effects (\eg flames, particles and cartoon characters) while strictly preserving spatial and temporal consistency. Video VFX editing is highly challenging because injected effects must blend seamlessly with the background, the background must remain entirely unchanged, and effect patterns must be learned efficiently from limited paired data. However, existing video editing models fail to satisfy these requirements. IC-Effect leverages the source video as clean contextual conditions, exploiting the contextual learning capability of DiT models to achieve precise background preservation and natural effect injection. A two-stage training strategy, consisting of general editing adaptation followed by effect-specific learning via Effect-LoRA, ensures strong instruction following and robust effect modeling. To further improve efficiency, we introduce spatiotemporal sparse tokenization, enabling high fidelity with substantially reduced computation. We also release a paired VFX editing dataset spanning $15$ high-quality visual styles. Extensive experiments show that IC-Effect delivers high-quality, controllable, and temporally consistent VFX editing, opening new possibilities for video creation.

• The paper introduces a robust, instruction-guided diffusion transformer framework that achieves precise and computationally efficient video VFX editing.
• It employs specialized token concatenation, causal attention, and a two-stage LoRA training regime to maintain spatiotemporal fidelity and accurate effect localization.
• Experimental evaluations demonstrate state-of-the-art performance on CLIP similarity metrics and human preference studies, enabling flexible, compositional multi-effect editing.

IC-Effect: An Instruction-Guided Diffusion Transformer Framework for Video VFX Editing

Overview and Motivation

IC-Effect introduces a precise and computationally efficient framework for video VFX editing capable of synthesizing complex dynamic effects—such as flames, particles, and cartoon elements—into existing videos under fine-grained textual instruction. The system is predicated on two technical desiderata for VFX editing: (1) seamless, physically plausible integration of effects while strictly preserving the spatiotemporal integrity of the input video, and (2) robust learning of effect patterns from limited paired data. Prior art in text-guided video editing largely fails to enforce background invariance and effect localization, often introducing visible content drift or global style corruption. IC-Effect's architecture leverages the high-capacity token modeling of Diffusion Transformers (DiT) with tailored in-context conditioning, efficient tokenization, and a two-stage LoRA-based training regime to overcome these bottlenecks.

Methodology

DiT-Based In-Context Conditioning and Sparse Tokenization

IC-Effect treats the source video as a clean contextual condition, using a specialized token concatenation and attention masking protocol to prevent contamination of source information during generation. Specifically, both the source and target videos are encoded into latent spaces using a 3D VAE. The source is represented by temporally and spatially downsampled tokens ($Z_S^{\downarrow}$ and $Z_I$), while the edited video is represented as standard noisy latent tokens ($Z_T$). Tokens are concatenated for joint attention. Causal attention ensures that only $Z_T$ can attend to $Z_S^{\downarrow}$/$Z_I$, not vice versa, thus preserving the original content and eliminating leakage of noise into the conditional path.

Figure 1: The overall architecture and training paradigm of IC-Effect, illustrating contextual token integration and two-stage LoRA fine-tuning.

The spatiotemporal sparse tokenization greatly reduces the quadratic complexity of attention over long video sequences, retaining full source fidelity while maintaining tractable memory and compute overheads.

Two-Stage Training: General Editor Pretraining and Effect-Specific LoRA

IC-Effect is trained in two phases: (1) high-rank LoRA adaptation for general video instruction-based editing on a large paired dataset, and (2) specialized low-rank Effect-LoRA fine-tuning on limited, highly-aligned VFX pairs. The base editor thus inherits strong generalization and instruction-following capabilities, while Effect-LoRA modules efficiently capture the idiosyncrasies of specific VFX styles with minimal data and compute.

Spatiotemporal Position Correction

To address misalignment artifacts induced by token sparsification, IC-Effect incorporates a spatiotemporal position correction mechanism: each temporally sparse token is explicitly mapped to the target's temporal segments, and spatial tokens are anchored to the first frame, ensuring precise propagation of conditional information throughout the editing process.

Experimental Analysis

Dataset Design

A new, large-scale paired VFX dataset is constructed, spanning 15 effect categories and providing triplets of (source, target, text instruction) with high temporal and structural congruence. This resolves a critical bottleneck in fair, reproducible evaluation of VFX editing systems.

Quantitative and Qualitative Evaluation

On standard and VFX-specific video editing benchmarks, IC-Effect sets new SOTA across all metrics:

• Best CLIP Image Similarity (0.9786), CLIP-Text/ViCLIP-Text alignment, and VBench sub-metrics.
• Outperforms baselines (InsV2V, InsViE, VACE, Lucy Edit) in structural preservation and effect accuracy (GPT-4o scoring: 4.79 structure, 4.56 effect).

Ablation studies show that spatiotemporal sparse tokenization achieves nearly full-token performance with a dramatic reduction in compute cost, and that both causal attention and position correction are essential for artifact-free generation.

Figure 2: Video VFX Editing Results of IC-Effect, showing strict adherence to instruction and consistent background preservation.

Figure 3: Qualitative Comparison of Video Editing. IC-Effect achieves high consistency, effect localization, and superior adherence to prompt compared to competing edit models.

Human and LLM-Based Evaluation

A human preference study demonstrates statistically significant user preference for IC-Effect in terms of instruction alignment and content consistency, corroborated by GPT-4o LLM evaluations.

Figure 4: Human Preference Study. IC-Effect is robustly preferred by human raters over all baselines for both instruction following and fidelity.

Advanced Capabilities and Ablation

IC-Effect supports flexible instruction-based control and compositional multi-effect editing, controllable purely through text guidance, as further demonstrated in additional supplementary results.

Figure 5: Flexible Instruction-Based Control of Video VFX Editing Results.

Figure 6: Video Multi VFX Editing of IC-Effect. Multiple effects can be injected accurately according to compositional instructions.

Theoretical and Practical Implications

IC-Effect operationalizes vision in-context learning in video via transformer-based conditional diffusion, pushing the boundary of efficient, few-shot effect learning for long-horizon spatiotemporal data. The architecture's decoupling of effect adaptation (LoRA modularity) from base editing functionality, combined with its rigorous conditional masking and tokenization strategies, is likely to generalize well to other complex, minimally-invasive video editing tasks, including fine-grained retargeting, style fusion, and content-aware augmentations.

Practically, IC-Effect's efficiency and precision make personalized, high-fidelity VFX editing feasible in production, lowering the data and compute barriers previously associated with token-level video generative editing.

Limitations and Future Directions

The primary limitation remains reliance on high-quality, carefully-paired source/effect data for new styles. Future work should address zero-shot or reference-based effect transfer and investigate modular feature extraction for cross-domain effect generalization.

Conclusion

IC-Effect constitutes a significant advance in the field of instruction-driven video VFX editing, integrating context-aware DiT modeling, spatiotemporal sparse conditioning, and modular effect-specific LoRA adaptation into a unified, robust, and efficient edit pipeline. It establishes new quantitative and qualitative benchmarks, offers a valuable dataset, and provides a theoretically-plausible template for scalable, controllable, and high-fidelity video effect synthesis.

Reference: "IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning" (2512.15635)