Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts

Abstract: Interactive video segmentation models such as SAM2 have demonstrated strong generalization across diverse visual domains. However, under weak user supervision, for example, when sparse point prompts are provided on a single frame, their predictions often suffer from temporal instability, including flickering boundaries, object dropout, and inconsistent object extents across frames. These issues limit their reliability in downstream video understanding and control applications. In this paper, we propose an inference-time temporal probability smoothing method that improves the temporal stability of SAM2-based video segmentation without retraining or architectural modification. Our approach operates directly on per-frame segmentation probability maps and leverages optical-flow-based motion warping together with pixel-wise uncertainty estimates derived from segmentation entropy, and forward-backwards flow consistency. These signals are used to adaptively blend current-frame predictions with motion-aligned historical estimates, yielding temporally coherent segmentation outputs under weak prompts. We evaluate the proposed method on four diverse video sequences using a comprehensive set of frame-wise and temporal stability metrics, including motion-compensated IoU, boundary consistency, object persistence, and area volatility. Experimental results demonstrate consistent improvements in temporal stability over vanilla SAM2 inference while preserving spatial accuracy. The proposed framework is lightweight, model-agnostic, and well-suited for real-time, interactive video segmentation.

• The paper introduces ViperSAM, a post hoc method that integrates optical flow warping and adaptive uncertainty blending to mitigate temporal instability in SAM2 segmentation.
• It demonstrates statistically significant improvements, including +1.54% in Warped IoU, +14.45% in Boundary F-score, and +27.02% in Unified Stability Score on challenging video sequences.
• The approach is model-agnostic and lightweight, offering immediate deployment benefits for real-time video analytics, robotics, and interactive segmentation applications.

Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts

Introduction and Context

Promptable segmentation foundation models such as SAM2 have established a robust paradigm for interactive video object segmentation, excelling in spatial generalization across visual domains. However, under weak prompt regimes (e.g., a sparse point prompt on a single frame), these models exhibit pronounced temporal inconsistency: mask flicker, unstable boundaries, abrupt area changes, and frequent object dropout hinder their viability for downstream video analytics, robotics, and control applications. Architectural modifications or retraining with temporal modules are non-trivial in foundation models due to engineering constraints and limited access to model internals.

The paper "Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts" (2604.17115) addresses this with a fully post hoc method: an inference-time temporal probability smoothing framework ("ViperSAM") that augments SAM2 predictions with motion-based alignment and pixelwise adaptive uncertainty fusion—explicitly designed for black-box, prompt-driven settings.

Figure 1: Overview of the ViperSAM framework, which enhances a frozen SAM2 with flow-based warping and pixelwise uncertainty fusion for temporally stable masks.

Methodology

The core ViperSAM framework operates directly on per-frame probability maps $\{P_t(x)\}$ from SAM2, introducing two operators: motion-aligned probability propagation and adaptive uncertainty-driven blending. The execution comprises the following:

1. Optical flow-based probability warping: Each mask probability from $t-1$ is spatially propagated to $t$ by warping via estimated dense optical flow $F_{t-1\rightarrow t}$ (Farnebäck method). This enforces motion consistency, yielding a propagated prior $\tilde{P}_t(x)$ usable even when the prompt signal is absent from the current frame.
2. Pixelwise uncertainty estimation: Two uncertainty signals are estimated: segmentation uncertainty by pixelwise entropy of $P_t(x)$, and motion uncertainty by the forward-backward cycle-consistency error of the estimated flow. These signals together provide a per-pixel indicator of which source to trust—the model’s own prediction ($P_t(x)$) or the propagated temporal prior ($\tilde{P}_t(x)$).
3. Adaptive blending: A convex combination coefficient $K_t(x)$ is computed from the normalized uncertainties for each pixel. The final mask is $$\hat{P}_t(x) = K_t(x)P_t(x) + (1-K_t(x))\tilde{P}_t(x)$$, thresholded for binary segmentation.

This procedure is lightweight (adds no trainable parameters), model-agnostic, and can be layered atop any prompt-driven segmentation method with real-valued output.

Experimental Design

Experiments were conducted on four challenging video sequences (with 1080p resolution, varying object classes and motion profiles) under the single-point prompt regime. The model under test is a frozen large hierarchical SAM2. All per-frame inference, probability map post-processing, and metric computation use deterministic protocols with no hyperparameter adjustment per video.

Temporal stability is evaluated with a comprehensive suite of metrics:

• Warped IoU (motion-compensated mask overlap)
• Temporal IoU (raw overlap, no compensation)
• Boundary F-score (frame-to-frame contour consistency)
• Dropout fraction (object disappearance rate)
• Unified Stability Score (aggregate, robust normalization).

Statistical significance is assessed with paired nonparametric testing.

Warped IoU progression for challenging sequences is exemplified below:

Figure 2: Warped IoU over time for a rapid-motion ‘desert car’ sequence, indicating improved temporal coherence with ViperSAM.

Boundary consistency is tracked as follows:

Figure 3: Framewise boundary F-score during the ‘desert car’ sequence; temporal instability events are notably reduced with ViperSAM.

Raw temporal IoU is provided for completeness:

Figure 4: Temporal IoU for the same sequence, where ViperSAM suppresses abrupt mask fluctuations.

Robust summary using the Unified Stability Score is also provided:

Figure 5: Unified Stability Score for the ‘desert car’ sequence, emphasizing frame-level stabilization across all major failure modes.

Analogous analyses are performed for the ‘jumping lamb’ sequence, showing adaptive behavior under articulated deformation and self-occlusion:

Figure 6: Warped IoU for the ‘jumping lamb’ sequence demonstrates ViperSAM stability under nonrigid motion.

Figure 7: Temporal IoU over time for ‘jumping lamb’, with smoothing preventing catastrophic failures.

Figure 8: Boundary F-score trajectory for ‘jumping lamb’, reflecting improved contour smoothness on deforming shapes.

Figure 9: USS for ‘jumping lamb’, confirming consistent stabilization in aggregate.

Results

ViperSAM provides statistically significant improvements across all principal stability metrics. On average over four videos, improvements are as follows:

• Warped IoU: +1.54% absolute (baseline: 0.962 → ViperSAM: 0.977)
• Boundary F-score: +14.45% absolute (0.483 → 0.553)
• Temporal IoU: +2.67% absolute (0.906 → 0.930)
• Unified Stability Score: +27.02% absolute (0.412 → 0.524)

Improvements are robust to video content; the largest gains manifest in sequences with rapid or nonrigid motion, which are failure modes for vanilla SAM2. There is negligible computational overhead and no degradation in object persistence (dropout fraction remains zero).

Notably, boundary-level stabilization (+54% in some sequences) far exceeds region-level gains, highlighting that the smoothing method excels not merely at preventing large-scale mask dropout but also at mitigating high-frequency contour flicker that are paramount for downstream temporal reasoning or physical interaction.

Analysis and Implications

The results substantiate that inference-time, uncertainty-driven temporal smoothing can mitigate the canonical instability artifacts in prompt-driven segmentation without architectural access. This addresses a critical deployment bottleneck for black-box foundation models: their historical insensitivity to video-domain temporal constraints under sparse supervision.

Practically, ViperSAM’s lightweight nature and model-agnostic construction mean it can be instantly deployed for real-time interactive systems, or coupled with advanced flow estimation for higher-order temporal models. Theoretically, the framework formalizes a class of methods that operate at the probability level, where unsupervised cues (optical flow, entropy) are exploited for dynamic trust assignment—extensible to other modalities such as 3D or multimodal video segmentation.

Failure modes, such as ambiguous motion or rapid scene illumination changes, are handled gracefully via the adaptive fusion, defaulting to model prediction when flow is unreliable. Areas for future exploration include: (1) integration with learned optical flow (to handle pathological motion); (2) extension to multi-object and panoptic regimes; (3) blending with feedback mechanisms for interactive prompt refinement.

Conclusion

The presented approach achieves substantial temporal stabilization of SAM2 video segmentation under weak prompts through a black-box, inference-only, uncertainty-aware probability fusion paradigm. It significantly reduces flickering, suppresses contour jitter, and eliminates object dropout—without retraining, model modification, or extra annotation. This contribution expands the effective deployment space for foundation model video understanding, particularly in robotics, surveillance, and interactive analysis where prompt sparsity and reliability are critical. The paradigm of post hoc probability smoothing is poised for further exploitation in temporally constrained AI systems.