OMTSeg: Open-Vocabulary Panoptic Segmentation

• OMTSeg is an architecture for open-vocabulary panoptic segmentation that integrates vision and text by using a unified BEiT-3 backbone alongside a Mask2Former-style segmentation head.
• It employs deep cross-modal attention, spatial feature adapters, and prompt tuning to adapt pre-trained representations for dense prediction tasks.
• Empirical results on datasets like ADE20K and COCO validate OMTSeg's performance improvements, highlighting the effectiveness of its multiway fusion and visual adaptation strategies.

OMTSeg is an architecture for open-vocabulary panoptic segmentation that leverages large-scale vision-language pre-training, specifically using BEiT-3 as a unified multiway transformer backbone integrated with a Mask2Former-style segmentation head. OMTSeg enables generalized segmentation over arbitrary categories by fusing image and text input throughout the model, achieving state-of-the-art performance via deep cross-modal attention, spatial feature adapters, and prompt-tuned word tokens (Chen et al., 2024).

1. Unified Pipeline and Input Tokenization

An RGB image $I\in\mathbb{R}^{H\times W\times 3}$ and a list of category names form the model's inputs. The image is partitioned into $N_p$ non-overlapping patches (e.g., $16\times 16$ pixels each). Each patch is linearly embedded into a $d$-dimensional vector, resulting in $V\in\mathbb{R}^{N_p\times d}$, with a 1D positional encoding added to produce $V'$. For the language modality, all category names are concatenated into a marker-separated string (e.g., “$[WLS]$ person ; $[WLS]$ snow ; $[WLS]$ snowboard”), tokenized using a WordPiece tokenizer into $N_w$ tokens. The shared $N_p$0-dimensional embedding layer yields $N_p$1, and positional encodings produce $N_p$2. Since BEiT-3 was pre-trained on aligned multimodal data, $N_p$3 and $N_p$4 can be concatenated and directly processed by the subsequent transformer stack.

2. Layer-wise Multiway Fusion Transformer

OMTSeg directly inherits the BEiT-3 fusion encoder consisting of $N_p$5 identical multiway transformer layers (typically $N_p$6). Each layer ingests the concatenated sequence $N_p$7, applying multiway multi-head self-attention (MultiwayMHSA):

$N_p$8

with $N_p$9 attention heads (typically $16\times 16$0, $16\times 16$1), each head computing:

$16\times 16$2

By never segregating vision and language, cross-modal interaction is enabled at every transformer layer. After self-attention, $16\times 16$3 is split back into visual and language components, which are then separately updated via their respective feed-forward networks:

$16\times 16$4

The concatenated features $16\times 16$5 are propagated through all $16\times 16$6 layers, yielding deep, bidirectional cross-modal representations as opposed to CLIP's single-layer contrastive alignment.

3. Visual Adapters and Prompt Tuning

To address the absence of dense-prediction specialization in BEiT-3, OMTSeg introduces a frozen backbone with trainable “visual adapter” modules and a mechanism for category prompt tuning.

• Spatial Prior Module (SPM): A convolutional stem (ResNet-style) outputs multi-resolution feature maps at $16\times 16$7, $16\times 16$8, and $16\times 16$9 input scales; each is projected to $d$0 channels by $d$1 convolutions.
• Spatial Feature Injector (SFI): At each transformer block, the current patch features $d$2 are queries in a cross-attention with SPM outputs $d$3 as keys/values: $d$4, thereby enriching transformer tokens with local convolutional information.
• Multi-Scale Feature Extractor (MSFE): SPM features are attended back to updated transformer features: $d$5.
• Prompt Tuning: Only the $d$6 marker token embeddings are fine-tuned, with $d$7 for each $d$8 token, improving the alignment to category names.

A plausible implication is that spatial adaptation and targeted prompt tuning are critical in transferring BEiT-3's pre-trained representations to dense, open-vocabulary outputs.

4. Mask2Former-Style Multiway Segmentation Head

The final multi-modal representations yield input to a Mask2Former-style decoder:

• $d$9 (patch tokens) and $V\in\mathbb{R}^{N_p\times d}$0 (word tokens) from BEiT-3 and adapters are inputs.
• A fixed set of object queries $V\in\mathbb{R}^{N_p\times d}$1 (typically $V\in\mathbb{R}^{N_p\times d}$2) is initialized.
• Each of $V\in\mathbb{R}^{N_p\times d}$3 decoder layers performs:
  • Masked Multi-Head Cross-Attention (MaskedMHCA): Queries in $V\in\mathbb{R}^{N_p\times d}$4 attend to $V\in\mathbb{R}^{N_p\times d}$5 yielding $V\in\mathbb{R}^{N_p\times d}$6 object-centric features.
  • Multi-Head Cross-Attention with Language: Queries now also attend to $V\in\mathbb{R}^{N_p\times d}$7 for category semantics.
  • Self-Attention and Feed-Forward: Standard transformer sublayer updates.

Final outputs per query: (1) a binary mask logit map $V\in\mathbb{R}^{N_p\times d}$8, (2) a mask feature $V\in\mathbb{R}^{N_p\times d}$9, and (3) an objectness score $V'$0.

Panoptic fusion follows Mask2Father: instance masks are sorted by $V'$1, thresholded, assigned unique IDs, with “stuff” regions filled in via “stuff” category predictions.

5. Training Objectives and Optimization

The loss function combines three elements:

• Mask Loss: Per-pixel binary cross-entropy over the $V'$2 mask predictions:

$V'$3

• Open-Vocabulary Classification Loss: Cosine similarity between each $V'$4 and $V'$5 token embedding $V'$6, cross-entropy over correct category $V'$7:

$V'$8

• Contrastive Alignment Loss: CLIP-style contrastive loss to enhance alignment between (mask, word) pairs:

$V'$9

• The total loss is $[WLS]$0, with weights set to equal or tuneable scalars.

6. Model Hyperparameters

OMTSeg adopts hyperparameters mainly from BEiT-3 large:

Hyperparameter	Typical Value	Purpose
$[WLS]$1	$[WLS]$2	Embedding dimension
$[WLS]$3	$[WLS]$4	Transformer layers
$[WLS]$5	$[WLS]$6	Attention heads
$[WLS]$7	$[WLS]$8	FFN hidden dimension
$[WLS]$9	$[WLS]$0	Object queries
$[WLS]$1	$[WLS]$2	Decoder layers
Visual Adapter scales	$[WLS]$3, $[WLS]$4, $[WLS]$5	Multi-resolution feature injection
$[WLS]$6Embed	$[WLS]$7 per category	Prompt-tuning per token

These parameter selections support both large-capacity multimodal representation and dense spatial adaptation.

7. Empirical Results and Ablative Analyses

On ADE20K-150, OMTSeg achieves $[WLS]$8 mIoU, surpassing the prior best (ODISE) by over $[WLS]$9 points; on Pascal VOC-21 it reaches $[WLS]$0 mIoU. In panoptic segmentation, OMTSeg obtains PQ = $[WLS]$1, AP = $[WLS]$2, mIoU = $[WLS]$3 on ADE20K, outpacing ODISE's PQ = $[WLS]$4. On COCO Panoptic, OMTSeg achieves PQ = $[WLS]$5 with a model size of $[WLS]$6M parameters (Chen et al., 2024).

Ablation studies on ADE20K demonstrate component necessity:

Ablation	PQ Delivered	Effect
Full OMTSeg	$[WLS]$7	Baseline
No cross-modal attention	$[WLS]$8	$[WLS]$99.1 PQ
No visual adapter	$N_w$0	$N_w$118.7 PQ
No prompt tuning ($N_w$2Embed)	$N_w$3	$N_w$41.6 PQ

This demonstrates the criticality of (1) BEiT-3’s layer-wise multiway fusion, (2) spatial adapters for adapting to dense-prediction, and (3) learnable prompt embeddings for open-vocabulary panoptic segmentation.

• Open-Vocabulary Panoptic Segmentation Using BERT Pre-Training of Vision-Language Multiway Transformer Model (Chen et al., 2024)