Cross-View Anchor Prompt Generator (V²-Anchor)

• The paper introduces V²-Anchor, which overcomes spatial prompting challenges in cross-view segmentation by translating object masks into normalized anchor prompts.
• It employs DINOv3 feature extraction, dense correspondence matching, and greedy stratified sampling to robustly select geometrically accurate anchor points.
• This non-learnable module seamlessly integrates with SAM2, delivering efficient, real-time, and geometry-aware segmentation performance across diverse viewpoints.

The Cross-View Anchor Prompt Generator (V²-Anchor) is a module within the V²-SAM cross-view object correspondence framework, designed to enable coordinate-based prompting for SAM2 in challenging cross-view segmentation scenarios. By leveraging DINOv3 patch-level features, V²-Anchor establishes geometry-aware correspondences between disparate views (e.g., egocentric to exocentric), enabling precise spatial localization of objects across drastic viewpoint, scale, and appearance changes. V²-Anchor is non-learnable, requires minimal computational resources, and restores SAM2’s strengths in point-based mask prompting even across highly non-trivial cross-view conditions (Pan et al., 25 Nov 2025).

1. Objectives and Motivating Principles

V²-Anchor addresses the core limitation of direct spatial prompting in cross-view setups, where the same object projects to different pixel locations in query and target views due to large viewpoint and appearance variations. In typical single-view SAM2 usage, spatial prompts (e.g., points, boxes) are defined with respect to a single image’s coordinate frame. Transferring these directly between views fails due to inherent geometric distortions. V²-Anchor’s goal is to uncouple prompt generation from the original image’s spatial layout by translating query-view object masks into reliable coordinate prompts in the target view, thereby “unlocking” coordinate-based segmentation for cross-view correspondence tasks.

2. Module Architecture and Feature Processing Pipeline

The V²-Anchor pipeline consists of several sequential stages leveraging DINOv3 ViT-L/16 features:

• DINOv3 Feature Extraction: Both query ($I_q$) and target ($I_t$) images are resized (longer side to 1024 px, aspect ratio preserved) and passed through a pre-trained DINOv3 encoder, yielding patchwise embeddings: $\phi_q \in \mathbb{R}^{F \times H_q \times W_q}$ and $\phi_t \in \mathbb{R}^{F \times H_t \times W_t}$, with $F=1024$, a typical patch grid of $64 \times 64$ (for patch size 16).
• Foreground Mask Projection: The object mask $M_q$ is downsampled to the patch grid, identifying the set of “foreground” patch indices $$\Omega = \{ i \mid \text{patch}_i \subset \text{object} \}$$.
• Dense Feature Matching: For each $i \in \Omega$, cosine similarity is computed across all target patches to generate a dense heatmap $H_{ij}$, and the best target patch $j^*(i)$ is selected per query patch.
• Point Stratification: 2D image coordinates ($p_{q_i}$, $p_{t_{j^*}}$) are recovered for each correspondence. To eliminate redundancy and local correlation, greedy stratified sampling enforces a minimum pairwise distance $\tau$, typically set to 32 px.
• Canonical Coordinate Transformation: Coordinates are normalized via a linear map $\Pi$ to $[0,1]^2$ relative to target image size, providing compatibility with SAM2’s prompt encoder.
• Prompt Encoding: The normalized anchor points are packaged as a coordinate prompt vector and passed to SAM2.

3. Mathematical Formulation of Geometry-Aware Correspondences

The correspondence generation within V²-Anchor is described by the following equations:

1. Dense Correspondence Heatmap:

$$H_{ij} = \frac{ \phi(I_q)_i^\top \phi(I_t)_j }{ \|\phi(I_q)_i\|_2 \|\phi(I_t)_j\|_2 }$$

where $H_{ij} \in \mathbb{R}^{(H_qW_q) \times (H_tW_t)}$.

2. Best-Match Target Selection:

$j^*(i) = \underset{j}{\arg\max} \; H_{ij}$

3. Stratified Anchor Sampling:

$$P'_t = \{ p_t^k: \| p_t^k - p_t^\ell \|_2 > \tau, \forall \ell < k \}$$

with $\tau$ typically set to twice the patch size.

4. Coordinate Normalization:

$$\tilde{p} = \Pi(p) = \left( \frac{p_x}{W_t^{orig}}, \frac{p_y}{H_t^{orig}} \right)$$

so that $\tilde{p} \in [0,1]^2$.

4. End-to-End Anchor Point Generation Algorithm

The operational flow of V²-Anchor is as follows:

1. Extract patch features from query and target images with DINOv3.
2. Downsample the query mask to identify relevant foreground patches $\Omega$.
3. For each $i \in \Omega$, compute $H_{ij}$ and find $j^*(i)$.
4. Recover original-image coordinates for each correspondence.
5. Assemble correspondence pairs $C = \{(p_{q_i}, p_{t_{j^*}})\}$.
6. Greedily select target points via stratification threshold $\tau$ to form $P'_t$.
7. Normalize each $p \in P'_t$ to $[0,1]^2$ producing $\tilde{P}_t$.
8. Input $\tilde{P}_t$ to SAM2’s Prompt Encoder, yielding the anchor prompt $P_{\mathrm{anchor}}^{q2t}$.

5. Integration with SAM2 Prompting and Inference Pipeline

V²-Anchor’s output—sparse, high-confidence anchor points—interfaces with the native SAM2 prompting mechanism. SAM2’s mask decoder is conditioned on coordinate, box, or mask prompts encoded by its Prompt Encoder. V²-Anchor supplies a small set (typically 1–3) of spatially stratified, normalized points in the target view’s coordinate frame. These anchors direct the mask decoder to focus on the corresponding object’s spatial extent in the new view. The Anchor Expert simply combines the anchor prompt with the (frozen) pre-trained decoder, requiring no further parameter tuning or supervision. Empirical ablation indicates sparse anchors maximize IoU, while overpopulating with anchors introduces noise.

6. Implementation Characteristics and Hyperparameters

A summary of V²-Anchor’s design settings is presented below.

Component	Value/Setting	Notes
DINOv3 Encoder	ViT-L/16, ImageNet pre-trained	Patch size 16 px; 1024 px longest side
Patch Grid	Typically $64 \times 64$	For $1024 \times 1024$ input
Stratification $\tau$	32 px	$2 \times$ patch size
Num. Anchor Points	1–3 (optimal); max 5	Ablation shows sparse anchors best
Coord. Transform	Linear division by img dims	To $[0,1]^2$ SAM2-compatible
Params Added	0 (non-learnable)	~0.5GB per GPU for DINOv3 activations
Walltime	~15 ms per image pair	Efficient for real-time use

Notably, V²-Anchor does not introduce any trainable parameters or require learning-based adaptation. Its stateless architecture supports direct deployment and low-latency operation on standard GPU hardware.

7. Significance and Context Within Cross-View Segmenter Research

V²-Anchor constitutes the first practical solution to coordinate-based prompt transfer for cross-view mask segmentation in pre-trained SAM2. By leveraging DINOv3’s implicit encoding of both appearance and geometry, it enables robust object localization under severe geometric and photometric perturbation, without reliance on explicit 3D models or additional supervision. In the V²-SAM framework, the combination of V²-Anchor (geometry-aware anchor prompting) and V²-Visual (appearance-guided visual prompting) forms a multi-expert system, with selection governed by a Post-hoc Cyclic Consistency Selector for reliability. V²-SAM sets new performance benchmarks on Ego-Exo4D, DAVIS-2017, and HANDAL-X, demonstrating V²-Anchor’s effectiveness for both standard cross-view correspondence and robotic-ready scenarios (Pan et al., 25 Nov 2025). This suggests V²-Anchor is an enabling component for generalizable, prompt-driven segmentation in computational perception pipelines.