CAV-SAM: Video-Based Reference Segmentation

• The paper introduces CAV-SAM, which reinterprets reference segmentation as a pseudo video sequence to leverage SAM2’s native video object tracking capabilities.
• It employs DBST for semantic interpolation and TTGA for lightweight, test-time geometric fine-tuning, resulting in significant performance improvements over state-of-the-art methods.
• CAV-SAM circumvents intensive episodic meta-training by adapting each test episode individually, demonstrating robust cross-domain segmentation with gains exceeding 5% mIoU.

Correspondence As Video for SAM (CAV-SAM) is a reference segmentation framework introduced in "Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild" (Wang et al., 11 Aug 2025). It addresses the setting in which a reference image $I_r$ and its mask $M_r$ are given, and the task is to segment the corresponding object or class in a target image $I_t$. Its defining idea is to reinterpret the implicit correspondence between the reference and target as a pseudo video sequence, allowing SAM2—used as an interactive video object segmentation (iVOS) model—to operate on reference segmentation without the extensive episodic meta-training that dominates prior few-shot segmentation (FSS) and cross-domain FSS (CD-FSS) approaches (Wang et al., 11 Aug 2025). The method consists of two modules: Diffusion-Based Semantic Transition (DBST), which constructs a semantic transformation sequence, and Test-Time Geometric Alignment (TTGA), which performs lightweight test-time fine-tuning for geometric alignment. On widely used datasets, it reports segmentation performance improvements exceeding $5\%$ over state-of-the-art methods (Wang et al., 11 Aug 2025).

1. Problem setting and conceptual reframing

In reference segmentation, the input consists of a reference image $I_r$, its mask $M_r$, and a target image $I_t$. The objective is to segment in $I_t$ the corresponding object or class indicated by the reference pair (Wang et al., 11 Aug 2025). Existing methods predominantly rely on meta-learning to acquire class-agnostic or domain-agnostic correspondence, which entails large episodic meta-training and substantial data and computational overhead (Wang et al., 11 Aug 2025).

CAV-SAM replaces this training-heavy formulation with a test-time reformulation. Instead of treating the reference-target pair as two unrelated still images, it interprets them as a pseudo video sequence

$I_v^1, I_v^2, \ldots, I_v^{N_v},$

intended to represent a smooth transformation from the reference to the target (Wang et al., 11 Aug 2025). This reframing converts a reference segmentation pair into a video-like temporal sequence and then applies SAM2’s native iVOS machinery to the resulting sequence.

The rationale is architectural rather than merely heuristic. SAM2 already supports prompt propagation, temporal consistency, and instance tracking in video settings (Wang et al., 11 Aug 2025). CAV-SAM therefore does not introduce a new correspondence learner trained across tasks; it instead attempts to make each test episode sufficiently video-like that SAM2’s pre-existing temporal prior becomes applicable. This suggests a broader adaptation principle: downstream tasks can be aligned to a foundation model’s native capability by recasting task structure rather than by learning a new adaptation procedure from large meta-training corpora.

2. Why SAM2 can be adapted in this way

SAM2 is described as a recent extension of SAM for images and videos, with a memory-augmented architecture and support for interactive video object segmentation (Wang et al., 11 Aug 2025). If prompted on one frame, it can track and segment across a temporal sequence. CAV-SAM exploits this property by constructing a pseudo sequence whose intermediate frames encode the reference-target correspondence.

The paper identifies two main obstacles to this reuse of video priors (Wang et al., 11 Aug 2025). The first is semantic discrepancy: in video object segmentation, the same object instance is typically tracked over time, whereas in reference segmentation the target may belong to the same class but not the same instance. The second is geometric variation: reference and target images may differ substantially in shape, pose, scale, and viewpoint, unlike the smoother transitions found in natural video.

A raw two-frame baseline already performs strongly. Simply concatenating $I_r$ and $M_r$0 as a two-frame pseudo video and feeding the result to SAM2 yields an average 1-shot mIoU of $M_r$1 over four datasets (Wang et al., 11 Aug 2025). This baseline indicates that the basic reframing is viable, but it also exposes the limitations of a direct two-frame construction. CAV-SAM therefore augments the sequence with DBST to address semantic discrepancy and TTGA to address geometric variation.

3. Diffusion-Based Semantic Transition (DBST)

DBST is the module responsible for constructing the pseudo video and primarily targets semantic discrepancy (Wang et al., 11 Aug 2025). Its goal is to generate intermediate frames $M_r$2 that transition semantically from the reference to the target. These frames need not be visually perfect in a human-perceptual sense; they need to provide enough semantic continuity for SAM2’s temporal propagation mechanism to operate effectively.

The module is built on a diffusion-model-based image morphing idea based on DiffMorpher (Wang et al., 11 Aug 2025). The paper describes a forward noising process

$M_r$3

that gradually corrupts data $M_r$4 into Gaussian noise,

$M_r$5

and a backward denoising process

$M_r$6

that reconstructs samples from noise, parameterized by a network $M_r$7 (Wang et al., 11 Aug 2025). DDIM inversion is used to obtain latent noises $M_r$8 and $M_r$9 for the reference and target images.

DBST performs interpolation in two spaces. In LoRA parameter space, LoRA residuals are fitted to the reference and target,

$I_t$0

and fused by linear interpolation: $I_t$1 In latent noise space, the inverted latent noises are interpolated by spherical linear interpolation: $I_t$2 where $I_t$3 is the angle between the latent vectors (Wang et al., 11 Aug 2025).

For each interpolation ratio $I_t$4, DBST interpolates the LoRA parameters, interpolates the latent noise, and denoises $I_t$5 with the diffusion model parameterized by

$I_t$6

under a DDIM schedule, yielding an intermediate frame $I_t$7 (Wang et al., 11 Aug 2025). Repeating this process for multiple $I_t$8 values produces the pseudo video sequence. The implementation details specify $I_t$9, $5\%$0 uniformly chosen from $5\%$1 to $5\%$2 excluding endpoints, DDIM inversion with 20 steps, LoRA rank 16, LoRA learning rate $5\%$3, and 200 LoRA training steps (Wang et al., 11 Aug 2025).

DBST differs from vanilla DiffMorpher in its optimization target. DiffMorpher aims for human-perceptual image morphing, whereas CAV-SAM requires only a natural pseudo video with sufficient semantic continuity for segmentation (Wang et al., 11 Aug 2025). Refinement modules considered unnecessary for segmentation are removed, reducing inference cost significantly. This establishes a deliberate tradeoff: less visually polished intermediate frames, but enough temporal coherence for SAM2’s tracking behavior.

4. Test-Time Geometric Alignment (TTGA)

TTGA addresses geometric variation by adapting SAM2 at test time using only the reference image and an augmented copy (Wang et al., 11 Aug 2025). The method extracts reference image features

$5\%$4

and computes a class prototype vector from the reference mask $5\%$5 via masked average pooling: $5\%$6 For target or pseudo-frame features $5\%$7, cosine similarity with the prototype gives

$5\%$8

The final binary prediction is produced by Otsu thresholding,

$5\%$9

These operations formalize the prototype-based mechanism through which TTGA derives activation masks (Wang et al., 11 Aug 2025).

Only the FPN layer, described as the “neck” of the SAM2 image encoder, is fine-tuned (Wang et al., 11 Aug 2025). The adaptation data are restricted to $I_r$0 and an augmented pair $I_r$1; no target image is used during adaptation. This keeps the update lightweight and prevents direct target leakage.

The central training signal is Augmentative Cyclic Consistency (ACC) (Wang et al., 11 Aug 2025). First, $I_r$2 is computed from the original reference. Second, it is applied to the augmented image to obtain a predicted mask $I_r$3, with loss

$I_r$4

Third, the prediction $I_r$5 is used to derive an augmented prototype vector $I_r$6, which is then used to predict back on the original image, yielding $I_r$7 and the cyclic loss

$I_r$8

The final objective is

$I_r$9

The paper compares ACC with Augmentative Bi-directional Consistency (ABC), in which the augmented ground-truth mask $M_r$0 is used directly to compute the augmented prototype (Wang et al., 11 Aug 2025). Reported average 1-shot mIoU is $M_r$1 for DBST only, $M_r$2 for DBST plus TTGA with ACC, and $M_r$3 for DBST plus TTGA with ABC (Wang et al., 11 Aug 2025). The reported advantage of ACC is attributed to its more challenging and regularizing nature: it requires reconstruction through the model’s own prediction rather than direct reuse of the augmented ground truth.

After test-time fine-tuning, the refined prototype vector $M_r$4 is used to activate pseudo video frames $M_r$5, producing pseudo-labels $M_r$6 (Wang et al., 11 Aug 2025). These are then used as extra prompts for SAM2. The paper states that the first half of the frames are prompted, enabling SAM2 to propagate segmentation as the pseudo video proceeds. In operational terms, TTGA supplies geometry-aware prompts, while SAM2 supplies spatio-temporal propagation.

5. Experimental results and ablations

CAV-SAM is evaluated on the standard CD-FSS benchmark datasets FSS-1000, DeepGlobe, ISIC2018, and Chest X-ray (Wang et al., 11 Aug 2025). These datasets cover both natural and medical imagery, and the paper uses SAM2 tiny as the iVOS model. The TTGA configuration uses 100 steps per reference image, learning rate $M_r$7, and a cosine annealing schedule (Wang et al., 11 Aug 2025).

The main quantitative result is that CAV-SAM outperforms previous methods by about $M_r$8 mIoU on average without meta-training (Wang et al., 11 Aug 2025). Its reported average scores are $M_r$9 for 1-shot and $I_t$0 for 5-shot. The best prior SAM-based reference method listed is APSeg, with $I_t$1 for 1-shot and $I_t$2 for 5-shot, corresponding to gains of $I_t$3 and $I_t$4 mIoU, respectively (Wang et al., 11 Aug 2025).

Setting	1-shot mIoU	5-shot mIoU
CAV-SAM	64.06	69.14
APSeg	61.30	65.09

Per-dataset results for CAV-SAM are reported as follows (Wang et al., 11 Aug 2025).

Dataset	1-shot	5-shot
DeepGlobe	39.11	44.16
ISIC	50.36	59.11
Chest X-ray	86.97	88.91
FSS-1000	79.78	84.38

The largest gains are reported on Chest X-ray, which the paper interprets as evidence of strong cross-domain robustness (Wang et al., 11 Aug 2025). At the same time, the method is not best on DeepGlobe. The paper attributes this to a mismatch between SAM’s object-segmentation prior and the region segmentation nature of DeepGlobe (Wang et al., 11 Aug 2025). This is an important qualification: the framework is effective across domains, but its compatibility with the task depends in part on the alignment between SAM2’s object-centric prior and dataset structure.

Ablation studies isolate the contribution of the two main modules. Average 1-shot mIoU rises from $I_t$5 for the SAM2 baseline to $I_t$6 with DBST and $I_t$7 with DBST plus TTGA (Wang et al., 11 Aug 2025). The same trend is reported with DEVA: $I_t$8 for the baseline, $I_t$9 with DBST, and $I_t$0 with DBST plus TTGA (Wang et al., 11 Aug 2025). These results indicate that pseudo-video construction alone is beneficial, and that test-time geometric alignment adds a further measurable gain.

6. Heuristic alternatives, failure modes, and limitations

The paper explicitly evaluates simple heuristic pseudo-video generation strategies and reports that they fail to match the proposed construction (Wang et al., 11 Aug 2025). Two alternatives are tested: Mixup for semantic transition and affine augmentation for geometric variation. Their average 1-shot mIoU values are $I_t$1 for Mixup and $I_t$2 for affine augmentation, compared with $I_t$3 for plain concatenation and $I_t$4 for CAV-SAM (Wang et al., 11 Aug 2025). The reported conclusion is that merely creating synthetic transitions is insufficient; the sequence must resemble a natural video-like semantic evolution.

The framework also discusses behavior under semantic mismatch. When the reference and target classes do not match, CAV-SAM should not hallucinate a segmentation (Wang et al., 11 Aug 2025). The paper argues that DBST uses Stable Diffusion text embeddings to maintain semantic coherence, so inconsistent classes lead to a meaningless sequence, while TTGA prototype vectors remain inactive because the semantic signal is absent. This suggests a desirable failure mode for open-world or “in the wild” reference segmentation: absence of valid correspondence should suppress confident but misleading masks rather than force a match.

Several limitations are identified directly (Wang et al., 11 Aug 2025). First, DBST still incurs diffusion cost because it relies on diffusion inversion and LoRA fitting, even though the method reduces cost relative to full DiffMorpher. Second, the method is not universally optimal, with lower performance on region-based tasks such as DeepGlobe where SAM’s object-centric prior is less suitable. Third, the approach is somewhat sensitive to DBST output quality, although the paper states that it remains robust as long as the sequence is acceptable. These limitations delimit the practical operating regime of the method: the pseudo video must be semantically coherent enough for SAM2, and the downstream task must be sufficiently aligned with object-oriented segmentation behavior.

7. Position within reference segmentation research

CAV-SAM is situated against several groups of baselines: traditional few-shot segmentation methods including PGNet, PANet, CaNet, RPMMs, PFENet, RePRI, HSNet, and SSP; cross-domain few-shot segmentation methods including PATNet, IFA, DR-Adaptor, ABCDFSS, and PMNNet; and SAM-based reference segmentation methods including VRP-SAM and APSeg (Wang et al., 11 Aug 2025). It also includes the strong concatenation baseline using SAM2 directly.

Its primary methodological distinction is that it avoids meta-training and instead performs adaptation on the fly at test time using only the reference image $I_t$5, its mask $I_t$6, and an augmented reference pair $I_t$7 (Wang et al., 11 Aug 2025). In that sense, the method differs fundamentally from meta-learning-based reference segmentation: it does not learn “how to adapt” from many tasks during training, but adapts each episode individually.

The broader implication stated in the paper is that large vision models may be adapted to downstream tasks not only by meta-learning or full fine-tuning, but also by recasting task structure so that it matches a model’s native capability (Wang et al., 11 Aug 2025). In CAV-SAM, that native capability is video prompt propagation. A plausible implication is that similar recastings may be useful whenever a foundation model possesses strong priors in one modality or task format that can be induced artificially in another. Within the specific scope of reference segmentation, CAV-SAM establishes the pseudo-video perspective as a concrete alternative to heavy episodic meta-training, grounded in DBST for semantic transition and TTGA for geometric alignment.