Token–Mask Cycle Consistency (TMCC)

• Token–Mask Cycle Consistency (TMCC) is a bidirectional training objective that enforces alignment between latent token representations and high-resolution pixel masks.
• It applies cycle-consistency losses in both Token→Mask and Mask→Token directions to maintain robust spatial grounding during segmentation.
• Integrating TMCC into AnchorSeg improves cumulative IoU performance and prevents the model from overfitting solely to final mask predictions.

Token–Mask Cycle Consistency (TMCC) is a bidirectional training objective designed to enforce alignment between discrete token-level representations and high-resolution pixel masks in reasoning segmentation models. Introduced in the AnchorSeg framework, TMCC addresses the challenge of spatial grounding under complex, implicitly specified textual queries by ensuring that information at both the latent token grid and the mask grid is mutually consistent through cycle-consistency losses. TMCC constitutes a core innovation for bridging the gap between compact spatial priors and pixel-level supervision when mapping language-conditioned queries to segmentations (Qian et al., 20 Apr 2026).

1. Motivation for Token–Mask Cycle Consistency

AnchorSeg reformulates reasoning segmentation by predicting a spatial response vector $S$ over $N$ image tokens after encoding with a language–vision model (LMM). While these token-level responses provide a compact spatial prior, solely supervising the upsampled pixel mask can lead to suboptimal spatial alignment; the model may “cheat” by overfitting to the loss on the final mask prediction, neglecting consistency with its latent token representations. TMCC is designed to explicitly “close the loop” by enforcing consistency in both directions:

• Token→Mask: Ensures that the upsampled, continuous token response map matches the ground truth mask.
• Mask→Token: Enforces that the downsampled or aggregated ground truth mask, when projected back onto the token grid, matches the latent spatial responses.

This bidirectionality compels the model to maintain coherent spatial grounding across resolutions, aligning intermediate and final spatial predictions (Qian et al., 20 Apr 2026).

2. Representations and Transformations

TMCC leverages structured representations at both token and mask granularity:

• Image Tokens: After LMM encoding, the image is $I = \{ i_1, ..., i_N \}$, with $i_i \in \mathbb{R}^d$.
• Anchor Query: The model learns a vector $q_{anc} \in \mathbb{R}^d$, such that token-level responses $S = \{ s_1, ..., s_N \}$ are computed as $s_i = i_i^T q_{anc}$.
• Token Grid: The vector $S$ is reshaped into a $G \times G$ grid (typically $G=24$), and normalized to $N$0.
• Upsampled Token Map ($N$1): Using bilinear interpolation, $N$2 is upsampled to the mask resolution $N$3, resulting in $N$4.
• Downsampled Mask ($N$5): The ground-truth binary mask $N$6 is optionally smoothed with a Gaussian filter ($N$7), resized to the token grid resolution with nearest neighbor interpolation, vectorized, and denoted $N$8.

All interpolation and filtering steps utilize the same routines as for constructing the spatial prior in AnchorSeg and are implemented within the training loop.

3. Bidirectional Cycle-Consistency Losses

TMCC introduces two symmetric loss terms to enforce bidirectional alignment:

• Token→Mask (Forward) Loss:

$N$9

Here, $I = \{ i_1, ..., i_N \}$0 denotes the upsampled spatial prior, and $I = \{ i_1, ..., i_N \}$1 is the ground-truth mask after Gaussian smoothing.

• Mask→Token (Backward) Loss:

$I = \{ i_1, ..., i_N \}$2

Here, $I = \{ i_1, ..., i_N \}$3 is the original token-level response, $I = \{ i_1, ..., i_N \}$4 the downsampled mask.

• Total TMCC Loss:

$I = \{ i_1, ..., i_N \}$5

$I = \{ i_1, ..., i_N \}$6 and $I = \{ i_1, ..., i_N \}$7 are set to 2.0 and 4.0, respectively, for both subterms.

4. Integration into AnchorSeg’s Training Objective

TMCC is incorporated as an essential component in AnchorSeg’s joint loss: $I = \{ i_1, ..., i_N \}$8 Where:

• $I = \{ i_1, ..., i_N \}$9 is the autoregressive language modeling loss for the query bank.
• $i_i \in \mathbb{R}^d$0 is the standard mask decoder loss (BCE + Dice) applied to the final predicted mask $i_i \in \mathbb{R}^d$1.
• $i_i \in \mathbb{R}^d$2 is the cycle-consistency loss.

While explicit values for $i_i \in \mathbb{R}^d$3 are not reported, all experiments include $i_i \in \mathbb{R}^d$4 with comparable weighting to the other loss terms. The TMCC framework is evaluated in all ablation studies and production models in AnchorSeg (Qian et al., 20 Apr 2026).

5. Implementation and Architectural Considerations

TMCC mandates a consistent spatial upsampling/downsampling pipeline:

• Token Map Upsampling: $i_i \in \mathbb{R}^d$5 is normalized and reshaped as a grid, then upsampled (bilinear interpolation) to a long-side of 336 pixels, preserving aspect ratio, and finally cropped/padded to $i_i \in \mathbb{R}^d$6. PyTorch’s F.interpolate is used with align_corners=False, antialias=True.
• Mask Downsampling: The binary mask $i_i \in \mathbb{R}^d$7 is resized to the corresponding 336-pixel canvas, then downsampled to $i_i \in \mathbb{R}^d$8 using nearest neighbor interpolation and vectorized. A 2D Gaussian kernel (σ=7, kernel size 31, reflect padding) is then applied to produce a soft target for token-level supervision.

This design tightly integrates the token–mask mapping with AnchorSeg’s query-bank architecture, ensuring spatial information is reliably transferred between latent and pixel resolutions in both directions.

6. Empirical Impact of TMCC on Reasoning Segmentation

Ablation studies in AnchorSeg quantitatively demonstrate the importance of TMCC for reasoning segmentation, evaluated on the ReasonSeg validation set:

Configuration	gIoU (%)	cIoU (%)
Only spatial prior (P)	68.3	71.6
Add TMCC, no other queries	–	74.0
P + TMCC + contextual	68.3	75.9
Only $i_i \in \mathbb{R}^d$9	67.4	73.8
Only $q_{anc} \in \mathbb{R}^d$0	64.4	68.2
Both (full TMCC)	68.3	75.9

The bidirectional structure is found to be essential; using both $q_{anc} \in \mathbb{R}^d$1 and $q_{anc} \in \mathbb{R}^d$2 outperforms either half in isolation. Adding TMCC alone delivers a clear improvement in cumulative IoU, and the full system—including spatial prior, TMCC, and contextual queries—yields the best reported performance, with a multi-point gain in cIoU (Qian et al., 20 Apr 2026).

7. Context and Implications

TMCC provides a generalizable approach to reconciling multi-resolution representations in structured reasoning tasks. By forcibly linking token-level and mask-level predictions, TMCC mitigates the risk of degenerate solutions where only the output mask is optimized, ignoring the intermediate spatial encoding. This suggests that similar bidirectional consistency objectives may benefit other multi-resolution visual grounding or segmentation systems, especially those leveraging token-based intermediate states. A plausible implication is the broader applicability of cycle-consistency principles for aligning learned priors with dense pixel-level annotation in other domains, provided the pipeline for upsampling, downsampling, and smoothing can be robustly implemented. TMCC’s integration and empirical gains underscore its significance as a methodological advance for high-fidelity reasoning segmentation (Qian et al., 20 Apr 2026).