AnchorSeg: Anchored Segmentation Methods

• AnchorSeg comprises two distinct frameworks: a 2019 unsupervised video segmentation using a fixed first-frame anchor and a 2026 reasoning segmentation leveraging a learned anchor token in a query bank.
• The 2019 method replaces recurrent propagation with anchor diffusion, establishing dense long-range correspondences to improve mask accuracy and achieve competitive DAVIS-2016 results.
• The 2026 model decouples semantic reasoning from spatial localization using a structured query bank and token–mask cycle consistency, enhancing performance on reasoning and referring segmentation tasks.

AnchorSeg is a name used in arXiv literature for two distinct segmentation formulations that organize prediction around an explicit anchor representation. In "Anchor Diffusion for Unsupervised Video Object Segmentation" (Yang et al., 2019), the anchor is the first video frame, which is used to establish dense long-range correspondences for unsupervised video object segmentation. In the later "AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation" (Qian et al., 20 Apr 2026), the anchor is a segmentation token $q_{anc}$ inside an ordered language-grounded query bank, used to provide explicit spatial grounding for reasoning segmentation.

1. Nomenclature and scope

The shared label conceals a substantial methodological discontinuity. The 2019 work, also described as AnchorSeg (AD-Net), addresses unsupervised video object segmentation and is built around anchor diffusion between video frames. The 2026 work addresses reasoning segmentation and reformulates segmentation as structured conditional generation over image tokens, conditioned on language grounded query banks (Yang et al., 2019, Qian et al., 20 Apr 2026).

Usage of "AnchorSeg"	Primary task	Anchor object
AnchorSeg (AD-Net), 2019	Unsupervised video object segmentation	First frame as a global "anchor"
AnchorSeg, 2026	Reasoning segmentation	Segmentation anchor token $q_{anc}$

A central source of terminological confusion is that the word anchor refers to different entities. In the video formulation, the anchor is a fixed visual reference frame. In the reasoning-segmentation formulation, the anchor is a learned token that determines localization signals while contextual queries provide semantic modulation. This suggests a common design motif—explicit anchoring—but not a shared architectural lineage.

2. Anchor diffusion for unsupervised video object segmentation

The 2019 formulation begins from a critique of prevailing unsupervised VOS pipelines. Contemporary methods are characterized as relying on recurrent networks such as Conv-RNN, Conv-LSTM, and Conv-GRU, or on pre-computed optical flow. The paper argues that RNNs are limited by vanishing or exploding gradients and truncated back-propagation through time, while one-step optical flow accumulates drift and can fail on static or slowly moving foregrounds. It further notes that a strong static image segmenter such as DeepLabv3 can already achieve competitive results, motivating a reconsideration of how temporal dependencies are modeled (Yang et al., 2019).

AnchorSeg (AD-Net) replaces sequential propagation with a fixed global reference. The first frame $I_0$ is treated as the anchor, and a later frame $I_t$ is matched directly to that anchor without conditioning on intermediate frames. The backbone encoder is DeepLabv3 with ResNet-101, output stride 8, and last-layer output dimension $c=128$. From the anchor and current frame it extracts

$$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$

where each spatial location has a $c$-dimensional embedding obtained from bilinear-upsampled DeepLabv3 features.

The architecture contains three branches: an identity skip from $X_t$, an intra-frame non-local branch on $X_t$, and an anchor-diffusion branch between $X_0$ and $q_{anc}$0. Their outputs are concatenated and fused via a $q_{anc}$1 conv, LeakyReLU, dropout$q_{anc}$2, and a final $q_{anc}$3 sigmoid classification layer. The anchor-diffusion branch defines a dense correspondence matrix

$q_{anc}$4

with row-wise softmax. The scaling by $q_{anc}$5 follows "Attention Is All You Need" to avoid large dot-products pushing the softmax into low-gradient regimes. The diffused feature map is then

$q_{anc}$6

The intended effect is that pixels in the current frame that match foreground embeddings in the anchor receive stronger aggregate responses, while background distractors are attenuated.

The intra-frame non-local branch applies the same principle within a single frame:

$q_{anc}$7

This is described as improving single-frame accuracy through long-range spatial context, whereas anchor diffusion enforces temporal consistency.

3. Optimization, inference, and empirical profile of AD-Net

Training uses only per-pixel binary cross-entropy. If $q_{anc}$8 is the sigmoid output and $q_{anc}$9 is the ground-truth mask, the loss is

$I_0$0

The method explicitly avoids optical-flow loss, reconstruction loss, adversarial loss, and consistency losses. Training data consist of the DAVIS-2016 training set with 30 videos; each iteration samples $I_0$1 with $I_0$2 fixed as frame 1 and $I_0$3 as a random later frame. Augmentation uses random crops around foreground and random rotations by multiples of $I_0$4. Optimization uses SGD with weight decay $I_0$5, batch size 8, initial learning rate 0.005, "poly" decay $I_0$6, and 30k total iterations. The backbone is initialized from an ImageNet-pretrained DeepLabv3-ResNet101, with other layers random (Yang et al., 2019).

At inference, anchor features $I_0$7 are computed once per video. Multi-scale evaluation uses $I_0$8 together with horizontal flips; heatmaps are averaged and thresholded at 0.5 to obtain the final mask. An optional post-hoc "instance pruning" stage uses an off-the-shelf detector+tracker to remove small or static distractor instances.

The DAVIS-2016 ablation establishes the contribution of the anchor mechanism. A DeepLabv3 baseline reaches $I_0$9 and $I_t$0. Adding intra-frame non-local yields $I_t$1 and $I_t$2. Adding anchor concat yields $I_t$3 and $I_t$4. Using anchor-diffusion only yields $I_t$5 and $I_t$6. The full AD-Net at single scale reaches $I_t$7 and $I_t$8. On the DAVIS-2016 validation benchmark, AD-Net (single-scale) reports $I_t$9 and $c=128$0, while AD-Net + I.Prun reports $c=128$1 and $c=128$2, ranking first on the DAVIS-2016 leaderboard of unsupervised methods and remaining competitive with state-of-the-art online semi-supervised approaches. The paper also reports results competitive with the state of the art on FBMS and ViSal, and summarizes the system as running online at $c=128$3 fps without any per-video fine-tuning (Yang et al., 2019).

The stated limitations are specific to the fixed-anchor design. A single first-frame anchor can fail if the true object appears only later or is heavily occluded initially. Appearance changes beyond the representational capacity of fixed anchor embeddings can reduce correspondence quality. Complex multi-object scenarios with multiple similar instances may require instance-level reasoning beyond binary foreground/background. The paper identifies multi-anchor schemes, dynamic anchor weighting, additional motion or geometry cues, and joint instance embedding with anchor diffusion as possible extensions.

4. Language-grounded query banks for reasoning segmentation

The 2026 AnchorSeg addresses a different problem: reasoning segmentation, where the query may be implicit and multi-step, such as “the object that provides shade in this scene” or “the thing that could prevent water damage,” rather than an explicit referring expression. Existing LMM-based solutions are characterized as inserting a single special token $c=128$4 and using its hidden state as a unified query for a mask decoder. The paper identifies a bottleneck in this formulation: semantic reasoning and spatial localization are compressed into one vector, limiting explicit disentanglement of what to segment from where to segment (Qian et al., 20 Apr 2026).

AnchorSeg reformulates the problem as structured conditional generation over image tokens. Instead of a single segmentation embedding, it extends the LMM vocabulary with $c=128$5 latent reasoning tokens $c=128$6 plus one $c=128$7, producing the ordered query bank

$c=128$8

where $c=128$9 capture intermediate semantic states and $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$0 is the spatial anchor query. In contrast to the 2019 anchor frame, this anchor is token-level and is generated autoregressively by the LMM.

The model defines a factorized distribution over scalar spatial responses $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$1 for image tokens $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$2:

$$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$3

In practice, the spatial response is implemented by anchor-based similarity,

$$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$4

The resulting responses are reshaped into a coarse grid, upsampled, and projected into a dense spatial prior $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$5, which is injected into the visual feature map by

$$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$6

The final mask is produced by the SAM mask decoder conditioned on the ordered query bank with learned positional embeddings:

$$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$7

The paper emphasizes that although the similarity score uses $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$8, the preceding $$X_0 \in \mathbb{R}^{(h\,w)\times c}, \qquad X_t \in \mathbb{R}^{(h\,w)\times c},$$9 shape the generation of $c$0 through the LLM’s autoregression and thus modulate localization semantically.

5. Token–Mask Cycle Consistency, implementation, and results

A central component of the 2026 model is Token–Mask Cycle Consistency (TMCC), introduced to bridge token-level predictions and pixel-level supervision. The token-to-mask term upsamples $c$1 to image resolution and supervises it against a smoothed ground-truth mask $c$2 with BCE and Dice:

$c$3

The mask-to-token term downsamples the smoothed mask to token resolution and aligns it with $c$4:

$c$5

The TMCC objective is

$c$6

and the overall training loss is

$c$7

The implementation couples an LLaVA 1.5 multimodal LLM, either 7B or 13B, with LoRA adapters ($c$8 for 7B and $c$9 for 13B), a CLIP-ViT-L/14@336 upstream encoder, and a ViT-H SAM encoder/decoder. Input images are resized so the long side is 336, padded to square for CLIP, and padded to 256 for SAM. The image token grid has $X_t$0 tokens of dimension $X_t$1. Query bank length is $X_t$2, with best performance reported at $X_t$3, corresponding to $X_t$4 latent tokens plus one anchor. After computing $X_t$5 at resolution $X_t$6, the method interpolates to $X_t$7 and applies a small conv head $X_t$8 channels to obtain $X_t$9. Optimization uses AdamW with $X_t$0, learning rate $X_t$1 for 7B or $X_t$2 for 13B, 100 warmup steps, weight decay 0, and grad-clip 1.0. Batch size is 2 samples per GPU with gradient accumulation 10 for effective batch size 20. The training mixture is ReasonSeg : ReferSeg : SemSeg = 1:1:1 over total $X_t$3 K images. Training runs for 120 epochs on 7B or 30 epochs on 13B, with 10 K steps per epoch. Loss weights are $X_t$4, $X_t$5, $X_t$6, $X_t$7, and $X_t$8 (Qian et al., 20 Apr 2026).

Inference proceeds in four stages: autoregress generation of the $X_t$9 latent reasoning tokens and then $X_0$0 to obtain $X_0$1; extraction of image tokens $X_0$2 and computation of $X_0$3, followed by normalization, reshaping, upsampling, and the conv head to form the spatial prior $X_0$4; SAM encoding of the image to produce $X_0$5 and injection of the prior by $X_0$6; and SAM mask decoding with cross-attention over the ordered query bank. No further post-processing is required beyond thresholding $X_0$7 at 0.5.

On the ReasonSeg test set, AnchorSeg-13B reports $X_0$8 gIoU and $X_0$9 cIoU. The nearest competitor listed in the summary, Seg-ReSearch-8B, reports $q_{anc}$00 gIoU and $q_{anc}$01 cIoU. AnchorSeg-7B reports $q_{anc}$02 gIoU and $q_{anc}$03 cIoU, compared with RSVP-GPT at $q_{anc}$04. On RefCOCO(+/g) [email protected], AnchorSeg-7B reaches up to $q_{anc}$05 on testA RefCOCO, and AnchorSeg-13B reaches up to $q_{anc}$06 on testA RefCOCO. On GRES, AnchorSeg-7B reports validation cIoU $q_{anc}$07 and null-target accuracy $q_{anc}$08. Ablations on ReasonSeg validation show that query bank length $q_{anc}$09 is best, with gIoU $q_{anc}$10 and cIoU $q_{anc}$11. Component ablations report cIoU $q_{anc}$12 without spatial prior $q_{anc}$13 or TMCC, $q_{anc}$14 with TMCC alone, $q_{anc}$15 with $q_{anc}$16 alone, $q_{anc}$17 with $q_{anc}$18 but no $q_{anc}$19, and $q_{anc}$20 for the full method. TMCC-specific ablations report $q_{anc}$21 cIoU for $q_{anc}$22 only, $q_{anc}$23 for $q_{anc}$24 only, and $q_{anc}$25 for both. Reported cost is training step latency $q_{anc}$26 s and 29.9 GB GPU; inference is $q_{anc}$27 FPS on A100.

6. Comparative interpretation and recurring themes

The two AnchorSeg formulations belong to different problem settings, use different backbones, and define anchors at different representational levels. The 2019 method uses DeepLabv3 with ResNet-101 and a fixed anchor frame for unsupervised VOS; the 2026 method uses LLaVA 1.5, CLIP-ViT-L/14@336, and ViT-H SAM, and defines an anchor token inside a language-conditioned query bank for reasoning segmentation (Yang et al., 2019, Qian et al., 20 Apr 2026).

The conceptual parallel is nevertheless precise. In the video formulation, anchor diffusion is introduced to avoid short-term temporal dependencies and drift from recurrent or flow-based propagation. In the reasoning formulation, the query-bank design is introduced to avoid overloading a single $q_{anc}$28 token with both semantic reasoning and spatial localization. This suggests a shared methodological preference for explicit factorization: long-range visual correspondence is separated from sequential propagation in the former, while spatial grounding is separated from semantic reasoning in the latter.

The limitations also differ in ways that reflect those design choices. The video model is vulnerable when the first frame is a poor reference, when appearance changes exceed the anchor embedding’s capacity, or when multiple similar instances require instance-level reasoning. The reasoning model is limited by LMM quality, the use of a fixed $q_{anc}$29 for all queries, and additional compute induced by multiple queries, the spatial prior, and TMCC. These constraints indicate that the term AnchorSeg identifies a family of anchoring strategies rather than a single canonical algorithm.