ANYPORTAL: Zero-Shot Video Editing Framework

• ANYPORTAL is a diffusion-based framework that unifies video and image models for joint background replacement, relighting, and foreground preservation.
• It employs a three-stage synthesis pipeline—background generation, light harmonization, and consistency enhancement—to ensure temporal coherence and accurate identity retention.
• The novel Refinement Projection Algorithm (RPA) deterministically re-encodes refined frames, mitigating reconstruction errors and ensuring pixel-level detail fidelity.

ANYPORTAL is a zero-shot, training-free framework for consistent video background replacement and foreground relighting. Given a foreground video $\mathbf{I}$ and a background prompt $p$ supplied as text or as an image, it seeks to generate a new video in which the background is replaced, the foreground subject is preserved in identity, details, and motion, the lighting is matched to the new environment, and the result remains temporally coherent across frames. Its central design is the composition of frozen pretrained diffusion components—a video diffusion model $\epsilon_\theta$ and IC-Light image diffusion models $\delta_p$ and $\delta_I$—together with a Refinement Projection Algorithm (RPA) that enforces pixel-level foreground consistency during video diffusion sampling (Gao et al., 9 Sep 2025).

1. Problem formulation and motivation

ANYPORTAL addresses video background replacement as a joint spatiotemporal editing problem rather than a framewise image-editing problem. The target output must satisfy four constraints simultaneously: background generation consistent with camera motion, foreground relighting that blends the subject into the new scene, temporal consistency over all frames, and pixel-level foreground preservation so that the subject does not drift, deform, or change identity (Gao et al., 9 Sep 2025).

The method is motivated by specific deficiencies in prior model classes. Image diffusion models such as IC-Light can perform background replacement and relighting, but because they operate frame by frame they suffer from temporal inconsistency in video. Video diffusion models capture real motion and temporal priors, but they are not naturally designed for precise foreground preservation or fine-grained background replacement. Existing video editing and control methods are described as providing only coarse control, such as edges, poses, or motion, rather than pixel-level identity preservation. The paper also argues that training or finetuning on video data is costly and impractical because paired video datasets for this task are scarce, while the compact 3D latent space of video diffusion models makes inversion and detailed latent manipulation difficult (Gao et al., 9 Sep 2025).

This problem framing places ANYPORTAL between single-frame relighting systems and generic video editors. It is neither a simple compositing pipeline nor a task-specific trained video model. “Zero-shot” in this context does not mean the absence of learned models; it means that no new model is trained and no base model is fine-tuned for the target task. “Training-free” likewise does not mean the absence of algorithmic intervention, since the framework relies on inference-time sampling, cross-frame attention, edge control, and RPA (Gao et al., 9 Sep 2025).

2. Three-stage synthesis pipeline

ANYPORTAL is organized into three stages: Background Generation, Light Harmonization, and Consistency Enhancement (Gao et al., 9 Sep 2025).

Stage	Main components	Output
Background Generation	$\delta_p$, DAS, ProPainter	$\mathbf{I}_b$
Light Harmonization	BiRefNet, $\delta_I$, $\delta_p$, cross-frame attention	$\mathbf{I}_L$
Consistency Enhancement	$p$0, edge-based ControlNet, RPA	final video

In the first stage, the system generates a background video $p$1 that matches the target prompt and the camera motion of the input video. Let $p$2 denote the first frame of $p$3. The text-guided IC-Light model is first applied as

$p$4

This relit first frame is then used by Diffusion-As-Shader (DAS), which conditions the video diffusion model on both $p$5 and the motion of the input video, represented as tracked 3D points or camera motion, to generate a coarse background video $p$6. Because this coarse output may still contain unwanted foreground remnants or mismatched objects, ProPainter is applied to remove the foreground and yield the final background video $p$7 (Gao et al., 9 Sep 2025).

In the second stage, the framework combines the original foreground subject with the new background and harmonizes illumination. BiRefNet is used to extract the foreground object $p$8. For each frame, the image-guided IC-Light model is applied:

$p$9

The paper distinguishes two behaviors: $\epsilon_\theta$0 gives better background consistency but weaker lighting effects, while $\epsilon_\theta$1 produces stronger relighting but poorer temporal consistency and background alignment. ANYPORTAL therefore applies an SDEdit-like refinement: $\epsilon_\theta$2 is noised for $\epsilon_\theta$3 steps and then denoised with the text-guided IC-Light model $\epsilon_\theta$4 conditioned on $\epsilon_\theta$5 and $\epsilon_\theta$6. The resulting output is a coarse relit video $\epsilon_\theta$7 (Gao et al., 9 Sep 2025).

In the third stage, the framework uses the temporal prior of the video diffusion model to improve coherence and then restores exact foreground details. Starting from $\epsilon_\theta$8, the method performs an SDEdit-style refinement with the video diffusion model $\epsilon_\theta$9 for $\delta_p$0 denoising steps. Edge-based ControlNet is used to preserve the rough structure of the original input video, while RPA supplies the missing pixel-level foreground fidelity (Gao et al., 9 Sep 2025).

3. Refinement Projection Algorithm

RPA is the core technical contribution of ANYPORTAL for foreground preservation. Its motivation is the instability introduced by naïve decode-edit-reencode cycles in the compact 3D latent space of video diffusion models. If one simply decodes a latent $\delta_p$1 to video, edits the foreground in pixel space, and re-encodes the result, reconstruction error and sampling randomness accumulate across DDIM steps, which can blur the background or distort details (Gao et al., 9 Sep 2025).

The method begins from the decoded video

$\delta_p$2

It then performs a frequency decomposition on both the edited video and the original input video:

$\delta_p$3

$\delta_p$4

A foreground mask is obtained by segmentation,

$\delta_p$5

and the background region is defined by inpainting the foreground from the current decoded video,

$\delta_p$6

The refined video is then composed as

$\delta_p$7

This construction keeps the edited frame’s low-frequency lighting and structure in the foreground while injecting the original video’s high-frequency details; the background is inherited from the inpainted edited frame (Gao et al., 9 Sep 2025).

The second half of RPA is the deterministic projection back into latent space. Standard VAE encoding is stochastic:

$\delta_p$8

but this sampling noise accumulates over denoising steps. RPA instead defines a deterministic projection direction using the decoded-and-reencoded original latent. If

$\delta_p$9

the algorithm sets

$\delta_I$0

and projects the refined result as

$\delta_I$1

The crucial alignment property is that if no refinement is applied, so that $\delta_I$2, then $\delta_I$3 exactly. This yields a deterministic re-encoding rule that preserves the original latent when no edit is made, while still allowing refined foreground details to influence the updated latent when refinement is active (Gao et al., 9 Sep 2025).

4. Sampling, attention, and control mechanisms

ANYPORTAL uses SDEdit-like noising and denoising in two distinct places: first in IC-Light harmonization, then in video diffusion refinement. In the harmonization stage, the noising depth $\delta_I$4 controls the strength of relighting: larger $\delta_I$5 yields stronger light and shadow changes, whereas smaller $\delta_I$6 yields more conservative editing. In the consistency enhancement stage, $\delta_I$7 controls the strength of video-diffusion refinement (Gao et al., 9 Sep 2025).

Cross-frame attention is introduced into both $\delta_I$8 and $\delta_I$9 to mitigate framewise variation. Instead of each frame using only its own self-attention features, all frames aggregate key and value features from the first frame. This is a direct mechanism for stabilizing appearance and illumination across time. The paper treats this as an important complement to the temporal prior of the video diffusion model: the IC-Light stage gains cross-frame appearance anchoring, while the video diffusion stage supplies broader temporal coherence (Gao et al., 9 Sep 2025).

The consistency enhancement stage uses DDIM sampling. The paper states the standard DDIM-style relation

$\delta_p$0

followed by the update

$\delta_p$1

Within this loop, RPA operates after decoding and before the update, thereby modifying $\delta_p$2 rather than changing the denoiser itself. Edge-based ControlNet is also applied at this stage, but the paper is explicit that edge control is only coarse structural guidance and cannot preserve exact identity details (Gao et al., 9 Sep 2025).

The framework introduces no new training loss because it is training-free. Instead, the paper distinguishes between optimization and evaluation. Fram-Acc, Tem-Con, ID-Psrv, and Mtn-Psrv are evaluation metrics, not losses used to fit model parameters. This distinction is central to the method’s identity: its novelty lies in composition and inference-time control rather than in supervised objective design (Gao et al., 9 Sep 2025).

5. Evaluation protocol, baselines, and reported results

ANYPORTAL is evaluated against three baselines: IC-Light per frame, TokenFlow, and DAS + IC-Light (Gao et al., 9 Sep 2025). The paper’s qualitative comparison positions these baselines along a recognizable trade-off frontier. IC-Light produces strong relighting but poor temporal consistency and may alter intrinsic foreground properties such as clothing colors. TokenFlow is described as having limited editing capability and weak foreground control. DAS + IC-Light improves motion-aware generation but remains weak in preserving foreground motion and detailed appearance (Gao et al., 9 Sep 2025).

The reported evaluation metrics are specialized to the task. Fram-Acc is a CLIP-based framewise prompt-alignment measure that tests whether the new background matches the target prompt better than the source prompt. Tem-Con measures CLIP similarity between consecutive frames and is used as a temporal consistency metric. ID-Psrv measures identity preservation using ArcFace loss on faces. Mtn-Psrv measures motion preservation using SpatialTracker point-tracking similarity (Gao et al., 9 Sep 2025).

According to the reported quantitative results, ANYPORTAL achieves best or near-best temporal consistency, best identity preservation, best motion preservation, and the highest user preference on temporal consistency, foreground preservation, and relighting quality. It attains second-best prompt alignment behind IC-Light, which the paper describes as expected because IC-Light is specialized for the single-frame prompt-alignment task (Gao et al., 9 Sep 2025).

The ablation studies are used to isolate the function of each module. Removing $\delta_p$3 weakens relighting. Removing consistency enhancement significantly degrades temporal consistency. Removing RPA worsens identity preservation, and the paper notes that naïve re-encoding makes the background blurry and degrades non-refined regions. Removing cross-frame attention increases style and appearance variation across frames. Increasing $\delta_p$4 strengthens lighting effects. Collectively, these ablations support the framework’s modular decomposition into motion-consistent background synthesis, relighting, and latent-space-preserving refinement (Gao et al., 9 Sep 2025).

6. Implementation profile, practical scope, and significance

The implementation reported in the paper uses CogVideoX as the video diffusion model and IC-Light as the image background replacement and relighting model. Sampling uses $\delta_p$5 steps. For illumination harmonization, the paper gives two operating regimes: a strong effect with $\delta_p$6 and a weaker effect with $\delta_p$7. Inference is performed on a single NVIDIA 4090 GPU, with CPU offload enabled for CogVideoX so that the system runs on a 24GB-class consumer GPU. Input videos are resized to $\delta_p$8, trimmed to 49 frames, and processed in about 12 minutes per video (Gao et al., 9 Sep 2025).

These implementation details define the paper’s practical scope. The framework is presented as usable on consumer-grade GPUs rather than requiring specialized multi-GPU training infrastructure. At the same time, its performance profile reflects the current constraints of pretrained video diffusion systems: moderate spatial resolution, fixed-length clips, and nontrivial per-video latency (Gao et al., 9 Sep 2025).

The paper frames the resulting capability as making “virtual teleportation” for video editing more practical: a person or object can be moved into a new scene, the background can be replaced by a text or image prompt, the foreground can be relit to match the new environment, motion can remain coherent, and details can be preserved without task-specific training (Gao et al., 9 Sep 2025). A plausible implication is that ANYPORTAL functions less as a monolithic model than as an overview pipeline over heterogeneous pretrained priors. This interpretation is reinforced by the paper’s statement that the method is modular and that newer pretrained diffusion models can be swapped in as they become available (Gao et al., 9 Sep 2025).

ANYPORTAL’s main significance lies in the way it reconciles three objectives that are usually separated across different systems: temporal coherence from video diffusion, relighting and harmonization from image diffusion, and pixel-level foreground preservation through deterministic latent projection. In the paper’s formulation, its central contribution is precisely this modular, training-free combination of IC-Light’s relighting ability, video diffusion’s temporal prior, and RPA for pixel-level foreground consistency in latent video diffusion (Gao et al., 9 Sep 2025).