---
title: 'CAV-SAM: Video-Based Reference Segmentation'
url: https://www.emergentmind.com/topics/correspondence-as-video-for-sam-cav-sam
type: topic
---

# CAV-SAM: Video-Based Reference Segmentation

Correspondence As Video for SAM (CAV-SAM) is a reference segmentation framework introduced in "Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild" [2508.07759]. It addresses the setting in which a reference image \(I_r\) and its mask \(M_r\) are given, and the task is to segment the corresponding object or class in a target image \(I_t\). Its defining idea is to reinterpret the implicit correspondence between the reference and target as a pseudo video sequence, allowing SAM2—used as an interactive video object segmentation (iVOS) model—to operate on reference segmentation without the extensive episodic meta-training that dominates prior few-shot segmentation (FSS) and cross-domain FSS (CD-FSS) approaches [2508.07759]. The method consists of two modules: Diffusion-Based Semantic Transition (DBST), which constructs a semantic transformation sequence, and Test-Time Geometric Alignment (TTGA), which performs lightweight test-time fine-tuning for geometric alignment. On widely used datasets, it reports segmentation performance improvements exceeding \(5\%\) over state-of-the-art methods [2508.07759].

## 1. Problem setting and conceptual reframing

In reference segmentation, the input consists of a reference image \(I_r\), its mask \(M_r\), and a target image \(I_t\). The objective is to segment in \(I_t\) the corresponding object or class indicated by the reference pair [2508.07759]. Existing methods predominantly rely on meta-learning to acquire class-agnostic or domain-agnostic correspondence, which entails large episodic meta-training and substantial data and computational overhead [2508.07759].

CAV-SAM replaces this training-heavy formulation with a test-time reformulation. Instead of treating the reference-target pair as two unrelated still images, it interprets them as a pseudo video sequence
\[
I_v^1, I_v^2, \ldots, I_v^{N_v},
\]
intended to represent a smooth transformation from the reference to the target [2508.07759]. This reframing converts a reference segmentation pair into a video-like temporal sequence and then applies SAM2’s native iVOS machinery to the resulting sequence.

The rationale is architectural rather than merely heuristic. SAM2 already supports prompt propagation, temporal consistency, and instance tracking in video settings [2508.07759]. CAV-SAM therefore does not introduce a new correspondence learner trained across tasks; it instead attempts to make each test episode sufficiently video-like that SAM2’s pre-existing temporal prior becomes applicable. This suggests a broader adaptation principle: downstream tasks can be aligned to a foundation model’s native capability by recasting task structure rather than by learning a new adaptation procedure from large meta-training corpora.

## 2. Why SAM2 can be adapted in this way

SAM2 is described as a recent extension of SAM for images and videos, with a memory-augmented architecture and support for interactive video object segmentation [2508.07759]. If prompted on one frame, it can track and segment across a temporal sequence. CAV-SAM exploits this property by constructing a pseudo sequence whose intermediate frames encode the reference-target correspondence.

The paper identifies two main obstacles to this reuse of video priors [2508.07759]. The first is semantic discrepancy: in video object segmentation, the same object instance is typically tracked over time, whereas in reference segmentation the target may belong to the same class but not the same instance. The second is geometric variation: reference and target images may differ substantially in shape, pose, scale, and viewpoint, unlike the smoother transitions found in natural video.

A raw two-frame baseline already performs strongly. Simply concatenating \(I_r\) and \(I_t\) as a two-frame pseudo video and feeding the result to SAM2 yields an average 1-shot mIoU of \(60.68\) over four datasets [2508.07759]. This baseline indicates that the basic reframing is viable, but it also exposes the limitations of a direct two-frame construction. CAV-SAM therefore augments the sequence with DBST to address semantic discrepancy and TTGA to address geometric variation.

## 3. Diffusion-Based Semantic Transition (DBST)

DBST is the module responsible for constructing the pseudo video and primarily targets semantic discrepancy [2508.07759]. Its goal is to generate intermediate frames \(I_v^1, I_v^2, \dots, I_v^{N_v}\) that transition semantically from the reference to the target. These frames need not be visually perfect in a human-perceptual sense; they need to provide enough semantic continuity for SAM2’s temporal propagation mechanism to operate effectively.

The module is built on a diffusion-model-based image morphing idea based on DiffMorpher [2508.07759]. The paper describes a forward noising process
\[
\{ q(\mathbf{z}_t) \mid t=0,1,\dots,T \}
\]
that gradually corrupts data \(q(\mathbf{z}_0)\) into Gaussian noise,
\[
q(\mathbf{z}_T) = \mathcal{N}(\mathbf{0}, \mathbf{I}),
\]
and a backward denoising process
\[
\{ p(\mathbf{z}_t) \mid t=T,T-1,\dots,0 \}
\]
that reconstructs samples from noise, parameterized by a network \(\varepsilon_\theta\) [2508.07759]. DDIM inversion is used to obtain latent noises \(\mathbf{z}_{Tr}\) and \(\mathbf{z}_{Tt}\) for the reference and target images.

DBST performs interpolation in two spaces. In LoRA parameter space, LoRA residuals are fitted to the reference and target,
\[
(\Delta \theta_r, \Delta \theta_t),
\]
and fused by linear interpolation:
\[
\Delta \theta_\alpha = (1-\alpha)\Delta \theta_r + \alpha \Delta \theta_t.
\]
In latent noise space, the inverted latent noises are interpolated by spherical linear interpolation:
\[
\mathbf{z}_{T\alpha} = \frac{\sin((1-\alpha)\phi)}{\sin \phi}\mathbf{z}_{Tr} + \frac{\sin(\alpha\phi)}{\sin\phi}\mathbf{z}_{Tt},
\]
where \(\phi\) is the angle between the latent vectors [2508.07759].

For each interpolation ratio \(\alpha\), DBST interpolates the LoRA parameters, interpolates the latent noise, and denoises \(\mathbf{z}_{T\alpha}\) with the diffusion model parameterized by
\[
\epsilon_{\theta+\Delta \theta_\alpha}
\]
under a DDIM schedule, yielding an intermediate frame \(I_v^i\) [2508.07759]. Repeating this process for multiple \(\alpha\) values produces the pseudo video sequence. The implementation details specify \(N_v = 9\), \(\alpha\) uniformly chosen from \(0.2\) to \(0.8\) excluding endpoints, DDIM inversion with 20 steps, LoRA rank 16, LoRA learning rate \(2\times 10^{-4}\), and 200 LoRA training steps [2508.07759].

DBST differs from vanilla DiffMorpher in its optimization target. DiffMorpher aims for human-perceptual image morphing, whereas CAV-SAM requires only a natural pseudo video with sufficient semantic continuity for segmentation [2508.07759]. Refinement modules considered unnecessary for segmentation are removed, reducing inference cost significantly. This establishes a deliberate tradeoff: less visually polished intermediate frames, but enough temporal coherence for SAM2’s tracking behavior.

## 4. Test-Time Geometric Alignment (TTGA)

TTGA addresses geometric variation by adapting SAM2 at test time using only the reference image and an augmented copy [2508.07759]. The method extracts reference image features
\[
F_r \in \mathbb{R}^{H\times W\times D}
\]
and computes a class prototype vector from the reference mask \(M_r\) via masked average pooling:
\[
\boldsymbol{p}_r = \mathrm{MAP}(F_r, M_r) = \frac{\sum_{i}^{H} \sum_{j}^{W} F_r[i,j,:]\cdot M_r[i,j]} {\sum_{i}^{H} \sum_{j}^{W} M_r[i,j]}.
\]
For target or pseudo-frame features \(F_t\), cosine similarity with the prototype gives
\[
S_t = \frac{F_t \cdot \boldsymbol{p}_r}{\|F_t\|_2 \|\boldsymbol{p}_r\|_2}.
\]
The final binary prediction is produced by Otsu thresholding,
\[
\hat{M}_t = \mathbb{I}(S_t > \tau), \quad \tau = \mathrm{otsu}(S_t).
\]
These operations formalize the prototype-based mechanism through which TTGA derives activation masks [2508.07759].

Only the FPN layer, described as the “neck” of the SAM2 image encoder, is fine-tuned [2508.07759]. The adaptation data are restricted to \(I_r, M_r\) and an augmented pair \(I_r^{\text{aug}}, M_r^{\text{aug}}\); no target image is used during adaptation. This keeps the update lightweight and prevents direct target leakage.

The central training signal is Augmentative Cyclic Consistency (ACC) [2508.07759]. First, \(\boldsymbol{p}_r\) is computed from the original reference. Second, it is applied to the augmented image to obtain a predicted mask \(\hat{M}_r^{\text{aug}}\), with loss
\[
\mathcal{L}_{\text{aug}} = \mathrm{BCE}(\mathrm{sigmoid}(S_r^{\text{aug}}), M_r^{\text{aug}}).
\]
Third, the prediction \(\hat{M}_r^{\text{aug}}\) is used to derive an augmented prototype vector \(\hat{\boldsymbol{p}}_r^{\text{aug}}\), which is then used to predict back on the original image, yielding \(\hat{M}_r\) and the cyclic loss
\[
\mathcal{L}_{\text{cyc}} = \mathrm{BCE}(\mathrm{sigmoid}(S_r), M_r).
\]
The final objective is
\[
\mathcal{L} = \mathcal{L}_{\text{aug}} + \mathcal{L}_{\text{cyc}}.
\]

The paper compares ACC with Augmentative Bi-directional Consistency (ABC), in which the augmented ground-truth mask \(M_r^{\text{aug}}\) is used directly to compute the augmented prototype [2508.07759]. Reported average 1-shot mIoU is \(62.08\) for DBST only, \(64.06\) for DBST plus TTGA with ACC, and \(61.54\) for DBST plus TTGA with ABC [2508.07759]. The reported advantage of ACC is attributed to its more challenging and regularizing nature: it requires reconstruction through the model’s own prediction rather than direct reuse of the augmented ground truth.

After test-time fine-tuning, the refined prototype vector \(\boldsymbol{p}_r\) is used to activate pseudo video frames \(I_v^1, I_v^2, \dots, I_v^{N_v}\), producing pseudo-labels \(\hat{M}_v^1, \hat{M}_v^2, \dots, \hat{M}_v^{N_v}\) [2508.07759]. These are then used as extra prompts for SAM2. The paper states that the first half of the frames are prompted, enabling SAM2 to propagate segmentation as the pseudo video proceeds. In operational terms, TTGA supplies geometry-aware prompts, while SAM2 supplies spatio-temporal propagation.

## 5. Experimental results and ablations

CAV-SAM is evaluated on the standard CD-FSS benchmark datasets FSS-1000, DeepGlobe, ISIC2018, and Chest X-ray [2508.07759]. These datasets cover both natural and medical imagery, and the paper uses SAM2 tiny as the iVOS model. The TTGA configuration uses 100 steps per reference image, learning rate \(1\times 10^{-3}\), and a cosine annealing schedule [2508.07759].

The main quantitative result is that CAV-SAM outperforms previous methods by about \(5\%\) mIoU on average without meta-training [2508.07759]. Its reported average scores are \(64.06\) for 1-shot and \(69.14\) for 5-shot. The best prior SAM-based reference method listed is APSeg, with \(61.30\) for 1-shot and \(65.09\) for 5-shot, corresponding to gains of \(+2.76\) and \(+4.05\) mIoU, respectively [2508.07759].

| Setting | 1-shot mIoU | 5-shot mIoU |
|---|---:|---:|
| CAV-SAM | 64.06 | 69.14 |
| APSeg | 61.30 | 65.09 |

Per-dataset results for CAV-SAM are reported as follows [2508.07759].

| Dataset | 1-shot | 5-shot |
|---|---:|---:|
| DeepGlobe | 39.11 | 44.16 |
| ISIC | 50.36 | 59.11 |
| Chest X-ray | 86.97 | 88.91 |
| FSS-1000 | 79.78 | 84.38 |

The largest gains are reported on Chest X-ray, which the paper interprets as evidence of strong cross-domain robustness [2508.07759]. At the same time, the method is not best on DeepGlobe. The paper attributes this to a mismatch between SAM’s object-segmentation prior and the region segmentation nature of DeepGlobe [2508.07759]. This is an important qualification: the framework is effective across domains, but its compatibility with the task depends in part on the alignment between SAM2’s object-centric prior and dataset structure.

Ablation studies isolate the contribution of the two main modules. Average 1-shot mIoU rises from \(60.68\) for the SAM2 baseline to \(62.68\) with DBST and \(64.06\) with DBST plus TTGA [2508.07759]. The same trend is reported with DEVA: \(49.57\) for the baseline, \(57.31\) with DBST, and \(59.93\) with DBST plus TTGA [2508.07759]. These results indicate that pseudo-video construction alone is beneficial, and that test-time geometric alignment adds a further measurable gain.

## 6. Heuristic alternatives, failure modes, and limitations

The paper explicitly evaluates simple heuristic pseudo-video generation strategies and reports that they fail to match the proposed construction [2508.07759]. Two alternatives are tested: Mixup for semantic transition and affine augmentation for geometric variation. Their average 1-shot mIoU values are \(52.21\) for Mixup and \(56.84\) for affine augmentation, compared with \(60.68\) for plain concatenation and \(64.06\) for CAV-SAM [2508.07759]. The reported conclusion is that merely creating synthetic transitions is insufficient; the sequence must resemble a natural video-like semantic evolution.

The framework also discusses behavior under semantic mismatch. When the reference and target classes do not match, CAV-SAM should not hallucinate a segmentation [2508.07759]. The paper argues that DBST uses Stable Diffusion text embeddings to maintain semantic coherence, so inconsistent classes lead to a meaningless sequence, while TTGA prototype vectors remain inactive because the semantic signal is absent. This suggests a desirable failure mode for open-world or “in the wild” reference segmentation: absence of valid correspondence should suppress confident but misleading masks rather than force a match.

Several limitations are identified directly [2508.07759]. First, DBST still incurs diffusion cost because it relies on diffusion inversion and LoRA fitting, even though the method reduces cost relative to full DiffMorpher. Second, the method is not universally optimal, with lower performance on region-based tasks such as DeepGlobe where SAM’s object-centric prior is less suitable. Third, the approach is somewhat sensitive to DBST output quality, although the paper states that it remains robust as long as the sequence is acceptable. These limitations delimit the practical operating regime of the method: the pseudo video must be semantically coherent enough for SAM2, and the downstream task must be sufficiently aligned with object-oriented segmentation behavior.

## 7. Position within reference segmentation research

CAV-SAM is situated against several groups of baselines: traditional few-shot segmentation methods including PGNet, PANet, CaNet, RPMMs, PFENet, RePRI, HSNet, and SSP; cross-domain few-shot segmentation methods including PATNet, IFA, DR-Adaptor, ABCDFSS, and PMNNet; and SAM-based reference segmentation methods including VRP-SAM and APSeg [2508.07759]. It also includes the strong concatenation baseline using SAM2 directly.

Its primary methodological distinction is that it avoids meta-training and instead performs adaptation on the fly at test time using only the reference image \(I_r\), its mask \(M_r\), and an augmented reference pair \(I_r^{\text{aug}}, M_r^{\text{aug}}\) [2508.07759]. In that sense, the method differs fundamentally from meta-learning-based reference segmentation: it does not learn “how to adapt” from many tasks during training, but adapts each episode individually.

The broader implication stated in the paper is that large vision models may be adapted to downstream tasks not only by meta-learning or full fine-tuning, but also by recasting task structure so that it matches a model’s native capability [2508.07759]. In CAV-SAM, that native capability is video prompt propagation. A plausible implication is that similar recastings may be useful whenever a foundation model possesses strong priors in one modality or task format that can be induced artificially in another. Within the specific scope of reference segmentation, CAV-SAM establishes the pseudo-video perspective as a concrete alternative to heavy episodic meta-training, grounded in DBST for semantic transition and TTGA for geometric alignment.

Source: https://www.emergentmind.com/topics/correspondence-as-video-for-sam-cav-sam