QueryWarp: Temporal Coherence in Video Diffusion

• The paper introduces QueryWarp, which improves temporal consistency in diffusion-based human motion video translation through inference-time query warping using pose-derived appearance flows.
• QueryWarp is a framework that modifies self-attention queries by fusing warped queries and current queries with occlusion masks to maintain structural fidelity across frames.
• Experimental results on human motion videos show enhanced editing accuracy and reduced pose distortion compared to traditional KV-only temporal consistency strategies.

Searching arXiv for QueryWarp and the directly named supporting methods to ground citations. QueryWarp is a framework for temporally coherent human motion video translation that modifies diffusion-based video editing by imposing temporal correlations directly on self-attention queries rather than relying only on cross-frame key and value sharing. The method is formulated for source videos of human actions and targets a translated video that follows the source motion while matching a target appearance or style specified by a text prompt, optionally with additional spatial conditions such as pose or edges. Its central mechanism is inference-time query warping: appearance flows extracted from source poses are used to warp the previous frame’s query token into the current frame, and an occlusion-aware fusion then replaces the current query in decoder self-attention. In the reported experiments, this yields higher temporal consistency while preserving local and structural regions more effectively than KV-only temporal consistency strategies (Zhu et al., 2024).

1. Problem formulation and motivation

QueryWarp addresses human motion video translation under the requirement that the output remain both temporally coherent and structurally faithful to the source motion. The stated setting takes a source human motion video $V=\{v_t\}$, a pose sequence $C=\{c_t\}$ extracted from that video, an optional edge condition such as HED boundaries, and a target text prompt $P^*$; the desired output is a translated video $V^*$ that preserves motion and semantics across frames while changing appearance or style according to the prompt (Zhu et al., 2024).

The motivation is tied to a specific limitation of frame-wise diffusion editing. When a text-to-image diffusion model such as Stable Diffusion is applied independently to each frame, even with pose or edge guidance, the result often exhibits flickering, geometry drift, and appearance changes across frames. Prior approaches therefore introduce temporal mechanisms, especially cross-frame propagation of key and value tokens in self-attention. QueryWarp identifies a remaining deficiency in that strategy: global style and colors may stabilize, but the query token remains frame-specific, and the query is described as governing spatial layout and structure. The paper therefore argues that local geometry can still change abruptly, including limb shape and clothing details, even when key/value sharing is present (Zhu et al., 2024).

This framing places QueryWarp in a narrow but technically important niche. It is not presented as a new diffusion training objective, nor as a new motion estimator; rather, it is an inference-time mechanism for constraining self-attention outputs across frames by coupling queries through explicit motion priors.

2. Architectural basis and conditioning pipeline

The framework uses Stable Diffusion 1.5 in latent space with ControlNet 1.0 for pose or edge conditioning, and the 2D U-Net is inflated to pseudo-3D following Tune-A-Video in order to enable temporal modeling. Sampling is performed with DDIM using 50 steps and classifier-free guidance scale 12. The experiments use source videos comprising 8 frames at $512 \times 512$ resolution (Zhu et al., 2024).

The latent diffusion formulation is written in the standard noise-prediction form. The forward diffusion process in latent space is

$$q(z_t \mid z_{t-1}) = \mathcal{N}(z_t; \sqrt{1-\beta_t}\, z_{t-1}, \beta_t I),$$

with training objective

$$\mathbb{E}_{z,\epsilon,t}\left[\|\epsilon-\epsilon_\theta(z_t,t,c_P,c_F)\|_2^2\right],$$

where $c_P$ is the text prompt embedding and $c_F$ is the ControlNet condition, such as pose or HED. DDIM sampling is written as

$$z_{t-1} = \sqrt{\alpha_{t-1}} \left( \frac{z_t - \sqrt{1-\alpha_t}\,\epsilon_\theta(z_t,t,c_P,c_F)}{\sqrt{\alpha_t}} \right) + \sqrt{1-\alpha_{t-1}}\,\epsilon_\theta(z_t,t,c_P,c_F).$$

These equations are part of the underlying diffusion model rather than a QueryWarp-specific loss or training criterion (Zhu et al., 2024).

The conditioning pipeline begins from human motion priors. Appearance flows $C=\{c_t\}$0 and occlusion masks $C=\{c_t\}$1 are predicted from pairs of poses using Dense Intrinsic Appearance Flow, denoted as

$C=\{c_t\}$2

The flow is pose-driven and is described as capturing foreground human motion while remaining agnostic to appearance or style changes between source and target. For small domain gaps, the framework can instead use optical flow between RGB frames via GMFlow, with occlusions computed from warp error; in that setting, HED boundaries are used as the ControlNet condition to stabilize edges (Zhu et al., 2024).

The architectural choice is therefore hybrid. Temporal structure comes from pseudo-3D denoising and cross-frame attention, but motion specificity comes from explicit pose-derived flow fields. This suggests that QueryWarp is designed to preserve motion semantics under larger source-to-target appearance gaps than approaches that depend on direct RGB correspondences.

3. Query warping and flow-guided self-attention

The defining operation of QueryWarp occurs in self-attention layers of the U-Net decoder. For a latent feature tensor $C=\{c_t\}$3 at a given layer, standard attention computes

$C=\{c_t\}$4

with

$C=\{c_t\}$5

Prior cross-frame approaches propagate an anchor frame’s keys and values,

$C=\{c_t\}$6

but leave $C=\{c_t\}$7 unchanged across frames (Zhu et al., 2024).

QueryWarp modifies precisely this term. Let $C=\{c_t\}$8 be the previous-frame query at decoder layer $C=\{c_t\}$9, reshaped to a spatial grid. The flow $P^*$0 is resized to that layer’s resolution and scaled to the appropriate token stride. QueryWarp then defines a backward warping operator $P^*$1 by bilinear sampling:

$P^*$2

or compactly,

$P^*$3

The warped query is fused with the current query using the occlusion mask:

$P^*$4

The semantics of the mask are explicit: where the previous frame remains valid or visible, the method propagates structure from the warped previous query; where occlusions or disocclusions occur, it reverts to the current query. The resulting attention replaces $P^*$5 with $P^*$6:

$P^*$7

The paper states that this imposes an implicit constraint on the self-attention output across frames because the attention maps are driven by a temporally consistent query that respects motion (Zhu et al., 2024).

Two design decisions are central here. First, query warping is applied at inference and introduces no additional training losses; no extra reconstruction, perceptual, or smoothness losses are used. Second, the mechanism is localized to decoder self-attention layers at multiple resolutions, while the encoder’s attention remains unchanged. This isolates the intervention to the part of the denoising process most directly tied to structure formation in the output frames (Zhu et al., 2024).

4. Inference procedure, variants, and workflow

The full procedure is organized as preprocessing followed by a denoising loop. Preprocessing extracts poses $P^*$8 from the source frames, computes $P^*$9 for $V^*$0, resizes flow and mask fields to the spatial grids of the relevant decoder attention layers at runtime, and chooses an anchor frame for key/value sharing. The paper describes the anchor as, for example, the first frame or a designated keyframe (Zhu et al., 2024).

During denoising, each frame latent is initialized either by DDIM inversion of the real frame or by random Gaussian noise. At every DDIM step and for every decoder self-attention layer, QueryWarp computes the current query $V^*$1. For frames with $V^*$2, it warps the previous frame’s query at the same denoising step and layer, fuses it with the current query through the occlusion mask, and then uses the fused query in cross-frame attention with the anchor frame’s $V^*$3 and $V^*$4. The modified attention output is then used in the remainder of the U-Net decoding path before the DDIM latent update is applied (Zhu et al., 2024).

The method is evaluated in both zero-shot and one-shot settings. In the zero-shot setting, QueryWarp uses pre-trained Stable Diffusion 1.5 and ControlNet 1.0 without fine-tuning on the source video. In the one-shot setting, denoted QueryWarp-One, the pseudo-3D U-Net is tuned following Tune-A-Video on the source sequence in order to strengthen temporal modeling, while the query warping mechanism itself remains unchanged. In both settings, the method is described as inference-time with respect to the query fusion operation (Zhu et al., 2024).

This division between zero-shot and one-shot is methodologically important. It implies that the reported gains are attributable to two separable factors: explicit query correlation through flow-guided warping, and optional source-specific temporal adaptation through Tune-A-Video-style tuning. The ablation results support this separation.

5. Empirical evaluation and ablation evidence

The evaluation uses 40 human motion videos, specifically dancing and sports clips, each manually captioned and represented as 8 frames at $V^*$5. Three metrics are reported: Editing Accuracy (Editing-Acc), defined as frame-wise semantic editing success where target prompt similarity exceeds source prompt similarity; Temporal Consistency (Tem-Con), defined as cosine similarity between consecutive frames; and Pose Distance (Pose-Dist), defined as the $V^*$6 distance between pose keypoints of source and translated videos (Zhu et al., 2024).

In zero-shot comparisons, the reported results are:

• TokenFlow: Tem-Con 0.9358, Editing-Acc 0.9417, Pose-Dist 32.1720
• FateZero: Tem-Con 0.9390, Editing-Acc 0.6154, Pose-Dist 22.0665
• Follow-Your-Pose: Tem-Con 0.9501, Editing-Acc 0.9640, Pose-Dist 64.4661
• QueryWarp: Tem-Con 0.9563, Editing-Acc 0.9429, Pose-Dist 28.5438

In one-shot comparisons, the reported results are:

• Tune-A-Video: Tem-Con 0.9575, Editing-Acc 0.8432, Pose-Dist 31.4616
• VideoP2P: Tem-Con 0.9575, Editing-Acc 0.6708, Pose-Dist 29.9871
• ControlVideo2: Tem-Con 0.9590, Editing-Acc 0.9428, Pose-Dist 31.0421
• QueryWarp: Tem-Con 0.9658, Editing-Acc 1.0000, Pose-Dist 29.4123

In the noise-initialization setting with HED conditions, the reported results are:

• Rerender A Video: Tem-Con 0.9648, Editing-Acc 0.6879, Pose-Dist 17.0119
• Text2Video-Zero: Tem-Con 0.9597, Editing-Acc 1.0000, Pose-Dist 16.3427
• ControlVideo1: Tem-Con 0.9741, Editing-Acc 1.0000, Pose-Dist 15.0137
• QueryWarp: Tem-Con 0.9697, Editing-Acc 1.0000, Pose-Dist 12.0414 (Zhu et al., 2024)

The ablation study isolates the contribution of query warping and of occlusion-aware fusion:

• Baseline (KV-only cross-frame attention): Tem-Con 0.9284, Editing-Acc 0.9053, Pose-Dist 35.9950
• w/o fusion (use warped query only): Tem-Con 0.9416, Editing-Acc 0.9217, Pose-Dist 30.1251
• QueryWarp-Zero: Tem-Con 0.9563, Editing-Acc 0.9429, Pose-Dist 28.5439
• QueryWarp-One: Tem-Con 0.9658, Editing-Acc 1.0000, Pose-Dist 29.4123

These results support two specific conclusions stated in the paper. First, query warping itself is essential: replacing KV-only attention with warped queries substantially improves temporal consistency and pose faithfulness. Second, fusion via the occlusion mask improves both temporal consistency and motion faithfulness relative to using the warped query alone. A plausible implication is that the mask is functioning as intended: it preserves propagated structure where motion correspondence is reliable and prevents error accumulation in occluded or newly revealed regions (Zhu et al., 2024).

Qualitatively, the framework is reported to preserve pose alignment and add new objects or styles reliably, including examples such as “a panda dancing with a moon background,” while reducing flicker and improving consistency in limbs, clothing, and hair.

6. Limitations, misconceptions, and technical significance

A recurrent misconception in diffusion-based video editing is that temporal consistency can be solved by sharing only keys and values across frames. QueryWarp explicitly contests that view by treating the query as the component that governs spatial layout and structure; under that interpretation, leaving queries frame-specific permits local structural instability even when style-level coherence is maintained. The ablation against the KV-only baseline operationalizes this claim (Zhu et al., 2024).

The framework is nevertheless bounded by the quality of its motion priors. Errors in appearance flow or occlusion mask prediction, particularly under complex self-occlusions or extreme fast motion, can reduce temporal coherence or introduce artifacts if the mask is inaccurate. Background motion is also not the primary target of the pose-driven flow model; it is handled by the current query branch and may still flicker if not constrained by additional conditions. These are stated limitations rather than inferred ones (Zhu et al., 2024).

The method’s efficiency profile is relatively modest. QueryWarp is reported to add minimal overhead for storing flows and masks and for performing bilinear sampling per attention layer, and it scales to 8 frames at $V^*$7 with 50 DDIM steps in the reported experiments. Code release is not specified in the text, although the framework is built on public Stable Diffusion and ControlNet components (Zhu et al., 2024).

In methodological terms, QueryWarp occupies a specific position in the literature on controllable video generation and editing. It does not alter the latent diffusion objective, and it does not require a new learned temporal regularizer. Instead, it inserts an explicit, motion-conditioned query prior into decoder self-attention. This suggests a broader design principle for diffusion-based video models: temporal coherence may depend not only on propagating content carriers such as keys and values, but also on explicitly constraining the query pathway that determines local spatial organization.