Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos (2511.19936v1)

Abstract: Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate their self-attention maps can be reinterpreted as semantic label propagation kernels, providing robust pixel-level correspondences between relevant image regions. Extending this mechanism across frames yields a temporal propagation kernel that enables zero-shot object tracking via segmentation in videos. We further demonstrate the effectiveness of test-time optimization strategies-DDIM inversion, textual inversion, and adaptive head weighting-in adapting diffusion features for robust and consistent label propagation. Building on these findings, we introduce DRIFT, a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement, achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks.

• The paper shows that pre-trained image diffusion models can achieve zero-shot video object tracking by repurposing self-attention for cross-frame label propagation.
• Test-time optimizations such as DDIM inversion, mask-specific textual inversion, and adaptive head weighting enhance temporal stability and segmentation quality.
• Integration with SAM for mask refinement in the Drift framework attains state-of-the-art performance on benchmarks like DAVIS-2016/2017 and YouTube-VOS.

Image Diffusion Models Enable Emergent Temporal Label Propagation for Video Object Tracking

Introduction

This paper rigorously characterizes the temporal reasoning capabilities of image-based diffusion models for the task of object tracking via video segmentation, introducing the Drift framework. The central claim is that pre-trained image diffusion models, although solely trained on image data, manifest emergent cross-frame correspondences through their self-attention mechanisms. When repurposed, these attention maps can be harnessed as zero-shot pixel-level propagation kernels, providing strong priors for maintaining semantic coherence and mask evolution across video frames, without requiring additional training or video data.

Temporal Label Propagation via Self-Attention

Diffusion models, when deployed for text-to-image synthesis, rely on attention modules that encode rich visual semantics for each pixel. The self-attention layers within these models implicitly capture intra-image correspondences and, crucially, enable the propagation of region masks informed by cross-frame visual similarity.

Propagation utilizes the attention maps to transfer labels from one frame to another, formalized as follows: given a query frame and a reference frame, attention maps are computed using learned query and key projections, with affinities subsequently aggregated across multiple heads. The mask for the target frame is updated by propagating the mask from the reference using the cross-frame attention kernel.

Figure 1: Input frames illustrate the propagation of object labels between video frames using cross-frame self-attention.

Empirical comparison with raw feature cosine similarity demonstrates the superiority of learned self-attention: cosine similarity disperses activations across irrelevant regions, while self-attention remains spatially localized.

Figure 2: Per-frame $\mathcal{J}$/$\mathcal{F}_{\mathrm{m}}$: Self-attention enables temporally consistent mask propagation, outperforming cosine similarity baselines.

Multi-head self-attention aggregates diverse semantic correspondences, further improving propagation accuracy and robustness compared to single-head approaches.

Test-Time Optimization Strategies

Label propagation via self-attention can be further enhanced using three complementary test-time optimization techniques:

• DDIM Inversion: Instead of injecting random Gaussian noise at large timesteps, DDIM inversion perturbs the input image using model-aligned noise, stabilizing semantics during propagation and enabling higher peak and sustained performance across timesteps.
• Mask-specific Textual Inversion: Rather than using generic or class-name prompts, textual inversion is employed to learn prompt tokens optimizing for self-consistent mask propagation. These learned embeddings are distinct from standard semantic tokens and act as fine-grained controllers over self-attention.

  Figure 3: t-SNE visualization reveals learned embeddings form distinct clusters, separated from class name embeddings.

• Adaptive Head Weighting: Test-time optimization assigns non-uniform weights to attention heads, prioritizing those that capture stronger semantic correspondences. Jointly optimizing head weights with prompt embeddings delivers superior mask propagation.

Drift: Diffusion-Based Video Object Tracking Framework

The Drift pipeline integrates temporal label propagation from image diffusion models with SAM-guided mask refinement. Given an initial mask in the first video frame, object identity is propagated through subsequent frames using cross-frame attention, followed by high-resolution mask refinement via the Segment Anything Model (SAM).

Figure 4: Overall Drift pipeline: DDIM inversion, mask-specific prompt learning, cross-frame attention aggregation, mask propagation, and SAM-guided refinement.

Multi-frame label propagation is employed for improved temporal stability, aggregating mask predictions from multiple reference frames. In multi-object settings, label propagation is performed independently for each instance and the background, yielding strong instance separation.

SAM refinement further sharpens predicted masks, leveraging sampled point prompts from the normalized soft mask to generate candidate masks. The best candidate is selected via soft IoU with the propagated mask, providing fine-grained boundary accuracy.

Empirical Results and Component Analysis

Drift achieves state-of-the-art zero-shot video object segmentation on major benchmarks (DAVIS-2016, DAVIS-2017, YouTube-VOS, Long Videos) without requiring video or segmentation training data. Against baselines such as STC, DINO, DIFT, and SAM-PT, Drift demonstrates superior region accuracy and temporal coherence.

Ablation analysis confirms that every component—DDIM inversion, mask-specific textual inversion, adaptive head weighting, and SAM refinement—contributes substantial improvements in mask quality and stability. Notably, the textual inversion and adaptive head weighting facilitate discriminative instance propagation, while DDIM inversion ensures semantic preservation.

Qualitative results depict the impact of ablation: removing components introduces spatial drift or semantic ambiguity, while the complete model maintains precise and temporally coherent instance segmentation across frames.

Figure 5: Qualitative comparison across model variants shows full Drift maintains consistent and precise instance masks over time.

Practical and Theoretical Implications

The findings establish that large-scale image diffusion models possess latent temporal propagation priors, accessible via their self-attention without any video supervision. This reframes the understanding of generative model representations, indicating that their denoising and synthesis objectives induce emergent abilities beyond static recognition. The pipeline demonstrates that zero-shot object tracking and segmentation in videos is achievable at high accuracy without dedicated spatiotemporal training, potentially diminishing the reliance on large video datasets for downstream tasks.

SAM-guided mask refinement interacts synergistically with temporally coherent propagation, delivering high-resolution boundaries once the core localization is reliable.

Limitations and Future Directions

A principal limitation is the computational overhead of prompt textual inversion, which remains a one-time cost per object per video. Addressing this through more efficient optimization or amortized learning strategies is open for future exploration. The framework’s generality is supported across different diffusion architectures, yet further work can investigate scalability to even longer sequences, diverse domains, and integration with video-trained priors.

The paper evidences the untapped potential of pre-trained image diffusion models in video understanding, suggesting transferability of attention kernels to a broader suite of temporal tasks, possibly including tracking, activity recognition, and dense correspondence estimation.

Conclusion

Image diffusion models, despite lacking video pretraining, support temporal label propagation through their attention mechanisms. The Drift framework exploits these properties with strategically optimized test-time adaptations and SAM refinement, achieving state-of-the-art zero-shot performance in video object segmentation. This work substantiates the capacity of generative models as universal dense correspondence engines, bridging static and temporal vision tasks without explicit spatiotemporal supervision.