Plain Mask Transformer (PMT)

• Plain Mask Transformer (PMT) is a segmentation architecture that combines a frozen Vision Transformer encoder with a lightweight Transformer decoder to support both image and video segmentation tasks.
• It relocates task-specific query processing into the decoder, enabling multi-task encoder sharing without sacrificing inference speed or accuracy.
• Empirical results show PMT achieves competitive segmentation quality while running up to 3× faster on images and 8× faster on video compared to finetuned models.

Plain Mask Transformer (PMT) is a segmentation architecture designed to preserve the simplicity and low latency of encoder-only designs while keeping a Vision Foundation Model (VFM) encoder frozen and shareable across tasks. Introduced in "PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders" (Cavagnero et al., 26 Mar 2026), PMT targets both image and video segmentation by placing a lightweight Transformer decoder, the Plain Mask Decoder (PMD), on top of frozen Vision Transformer (ViT) features. The model is positioned against recent VFM-based encoder-only methods such as EoMT and VidEoMT, which inject learnable queries into the ViT’s self-attention and finetune the entire encoder; PMT relocates task-specific query processing into the decoder and thereby restores multi-task encoder sharing (Cavagnero et al., 26 Mar 2026).

1. Conceptual Positioning

PMT is motivated by the deployment setting in which modern large-scale VFMs such as DINOv3 produce patch-token representations rich enough to support multiple downstream tasks from a single frozen backbone. In that setting, encoder finetuning undermines one of the principal practical advantages of VFMs: a shared encoder representation that can be reused across panoptic, instance, semantic, and video segmentation tasks without task-specific backbone updates (Cavagnero et al., 26 Mar 2026).

Within this framing, PMT can be understood as a reconciliation of two previously competing properties. On one side are encoder-only segmentation systems such as EoMT and VidEoMT, which offer architectural simplicity and speed but require encoder finetuning. On the other side are frozen-encoder systems that preserve backbone shareability but often incur additional complexity or latency. PMT addresses this trade-off by introducing the PMD, a fast Transformer-based segmentation decoder that operates on frozen VFM features while maintaining the generality of the encoder-only framework (Cavagnero et al., 26 Mar 2026).

A common misconception is that retaining a frozen encoder necessarily entails a decisive accuracy penalty relative to finetuned alternatives. The PMT results do not support such a blanket conclusion: on standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to $\sim 3\times$ faster, and on video segmentation it performs on par with fully finetuned methods while being up to $8\times$ faster than state-of-the-art frozen-encoder models (Cavagnero et al., 26 Mar 2026). This suggests that, given sufficiently strong frozen features, a substantial portion of task adaptation can be moved into a lightweight decoder without forfeiting the latency profile associated with encoder-only formulations.

2. Architectural Composition

PMT consists of two principal components: a frozen ViT encoder pre-trained at scale, and the PMD, a small stack of Transformer layers that jointly processes a fixed set of learnable object queries and multi-depth encoder features (Cavagnero et al., 26 Mar 2026).

The processing pipeline is specified as follows. An input image $I \in \mathbb{R}^{3 \times H \times W}$ is passed through a frozen ViT encoder with $L$ layers. Patch tokens $X^l \in \mathbb{R}^{D \times N}$ are extracted at several depths $l_j$, including the final layer $L$. For each selected depth, PMT applies final LayerNorm, then a trainable BatchNorm and a two-layer MLP; the resulting feature maps are summed across depths to obtain multi-scale patch tokens $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$. A fixed set of $K$ learnable queries $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$ is then concatenated with the patch tokens to form a sequence of length $8\times$0, which is processed by $8\times$1 Transformer layers in the PMD, typically with $8\times$2 (Cavagnero et al., 26 Mar 2026).

The decoder produces refined queries $8\times$3 and refined patch tokens $8\times$4. Prediction is split into a classification branch and a mask branch. Class logits are defined by

$8\times$5

while mask logits are obtained via a dot-product between a small MLP applied to $8\times$6 and the upsampled patch tokens $8\times$7:

$8\times$8

In the full formulation, $8\times$9 is $I \in \mathbb{R}^{3 \times H \times W}$0 reshaped and bilinearly upsampled, and the dot-product is taken between the per-query MLP output of dimension $I \in \mathbb{R}^{3 \times H \times W}$1 and each spatial feature vector in $I \in \mathbb{R}^{3 \times H \times W}$2 (Cavagnero et al., 26 Mar 2026).

The design is notable for preserving a plain sequence-processing interface: queries and patches are handled jointly, and the same decoder principle is stated to apply seamlessly to both image and video segmentation. A plausible implication is that PMT inherits some of the implementation regularity of encoder-only systems while relocating all task-specific adaptation into an independently trainable module.

3. Plain Mask Decoder Mechanics

The PMD mirrors the architecture of the last $I \in \mathbb{R}^{3 \times H \times W}$3 layers of EoMT’s finetuned encoder but operates on top of frozen features and is trained from scratch (Cavagnero et al., 26 Mar 2026). Each decoder block $I \in \mathbb{R}^{3 \times H \times W}$4 implements the residual-preconditioned Transformer updates

$I \in \mathbb{R}^{3 \times H \times W}$5

$I \in \mathbb{R}^{3 \times H \times W}$6

Here, $I \in \mathbb{R}^{3 \times H \times W}$7 denotes LayerNorm and $I \in \mathbb{R}^{3 \times H \times W}$8 denotes multi-head self-attention.

For an input sequence $I \in \mathbb{R}^{3 \times H \times W}$9 at layer $L$0, the decoder computes

$L$1

followed by scaled dot-product attention,

$L$2

All tokens—queries and patches—attend to each other. The paper explicitly identifies this as realizing query self-attention and cross-attention with patches in one operation (Cavagnero et al., 26 Mar 2026). Rotary position embeddings (RoPE) are applied on patch tokens, while queries receive no positional encoding.

A distinctive training-time mechanism is masked attention. At each layer $L$3, an intermediate mask $L$4 is predicted per query, and the attention weights from query positions to patch positions are masked so that attention focuses only on regions where $L$5. This masking is gradually phased out by a mask-annealing schedule, such that inference uses standard unmasked MHSA, which is stated to be compatible with FlashAttention (Cavagnero et al., 26 Mar 2026). This arrangement couples localized training guidance with inference-time efficiency.

The ablation evidence underscores the functional importance of the decoder itself. On COCO panoptic segmentation, stepwise removal from ViT-Adapter+Mask2Former shows that removing the Transformer decoder in an EoMT-style frozen setting causes PQ to collapse to $L$6 at $L$7 FPS, whereas building up from frozen EoMT with a Plain PMD yields PQ $L$8 at $L$9 FPS; adding lateral connections yields PQ $X^l \in \mathbb{R}^{D \times N}$0 at $X^l \in \mathbb{R}^{D \times N}$1 FPS; adding RoPE yields PQ $X^l \in \mathbb{R}^{D \times N}$2 at $X^l \in \mathbb{R}^{D \times N}$3 FPS (Cavagnero et al., 26 Mar 2026). These measurements indicate that PMD is not a marginal accessory but the core mechanism enabling segmentation quality with frozen features.

4. Training and Inference Regimen

The encoder remains frozen throughout training; only PMD parameters are updated, including Transformer layers, MLPs in the prediction heads, lateral adapters, and BatchNorms (Cavagnero et al., 26 Mar 2026). This training boundary is central to PMT’s intended role as a shareable-head architecture on top of a common VFM.

The supervision setup comprises cross-entropy for classification between predicted $X^l \in \mathbb{R}^{D \times N}$4 and ground-truth class, together with binary cross-entropy plus Dice loss for mask prediction between $X^l \in \mathbb{R}^{D \times N}$5 and the ground-truth mask (Cavagnero et al., 26 Mar 2026). Optimization uses AdamW. For image models, the reported configuration is batch size $X^l \in \mathbb{R}^{D \times N}$6, learning rate $X^l \in \mathbb{R}^{D \times N}$7, cosine annealing, $X^l \in \mathbb{R}^{D \times N}$8 epochs on COCO, $X^l \in \mathbb{R}^{D \times N}$9 epochs on ADE20K, and a $l_j$0-step linear warmup. For video models, the configuration is batch size $l_j$1 with a $l_j$2-frame window, learning rate $l_j$3, polynomial decay with power $l_j$4, a training schedule following CAVIS, and a $l_j$5-step warmup. Data augmentations follow EoMT/VidEoMT protocols, including random flip and scale jitter (Cavagnero et al., 26 Mar 2026).

Inference speed is measured on NVIDIA H100 with FlashAttention-2 and torch.compile, using batch size $l_j$6 and mixed precision (Cavagnero et al., 26 Mar 2026). For images, PMT-L with ViT-L runs at $l_j$7 FPS versus ViT-Adapter+Mask2Former at $l_j$8 FPS, corresponding to approximately $l_j$9 higher throughput. For video, PMT-L runs at $L$0–$L$1 FPS versus frozen-encoder baselines at $L$2–$L$3 FPS, yielding up to approximately $L$4 higher throughput (Cavagnero et al., 26 Mar 2026).

These choices clarify that PMT’s efficiency claims are not limited to algorithmic FLOP reductions. They are tied to an inference path that avoids training-time masking, uses standard unmasked MHSA at test time, and is explicitly aligned with high-performance attention kernels.

5. Empirical Results on Image Segmentation

On COCO val2017, PMT is compared against ViT-Adapter + Mask2Former in the frozen-encoder regime. At resolution $L$5, ViT-Adapter + Mask2Former with DINOv3 and a frozen pyramid uses $L$6M parameters, $L$7 GFLOPs, runs at $L$8 FPS, and attains PQ $L$9 and AP $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$0. PMT with DINOv3 and a frozen pyramid uses $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$1M parameters, $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$2 GFLOPs, runs at $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$3 FPS, and attains PQ $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$4 and AP $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$5 (Cavagnero et al., 26 Mar 2026). At $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$6, ViT-Adapter + M2F reaches PQ $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$7 at $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$8 FPS, while PMT reaches PQ $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$9 at $K$0 FPS, i.e., approximately $K$1 faster (Cavagnero et al., 26 Mar 2026).

On ADE20K val, ViT-Adapter + M2F with DINOv3 uses $K$2M parameters and $K$3 GFLOPs, runs at $K$4 FPS, and obtains mIoU $K$5. PMT uses $K$6M parameters and $K$7 GFLOPs, runs at $K$8 FPS, and obtains mIoU $K$9 (Cavagnero et al., 26 Mar 2026). The paper summarizes this as PMT being more than $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$0 faster with less than $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$1 mIoU drop.

Benchmark	Baseline result	PMT result
COCO val2017 @ $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$2	PQ $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$3, AP $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$4, $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$5 FPS	PQ $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$6, AP $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$7, $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$8 FPS
COCO val2017 @ $$Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$$9	PQ $8\times$00, $8\times$01 FPS	PQ $8\times$02, $8\times$03 FPS
ADE20K val	mIoU $8\times$04, $8\times$05 FPS	mIoU $8\times$06, $8\times$07 FPS

The image results characterize PMT as a frozen-encoder model that remains very close to the strongest reported frozen-encoder baselines in quality while materially shifting the speed-accuracy operating point (Cavagnero et al., 26 Mar 2026). A plausible implication is that, for applications constrained by throughput or latency rather than by the last fraction of a point in PQ or mIoU, PMT changes the practical baseline for frozen-VFM segmentation.

6. Empirical Results on Video Segmentation

On YouTube-VIS 2019/2021 at $8\times$08, CAVIS with DINOv3 and a frozen encoder uses $8\times$09 GFLOPs, runs at $8\times$10 FPS, and attains AP19 $8\times$11, AP$8\times$12 $8\times$13, and AR$8\times$14 $8\times$15. PMT with DINOv3 and a frozen encoder uses $8\times$16 GFLOPs, runs at $8\times$17 FPS, and attains AP19 $8\times$18, AP$8\times$19 $8\times$20, and AR$8\times$21 $8\times$22 (Cavagnero et al., 26 Mar 2026). The reported summary is that PMT is approximately $8\times$23 faster while slightly improving AP.

On VIPSeg val, CAVIS with DINOv3 and a frozen encoder uses $8\times$24 GFLOPs, runs at $8\times$25 FPS, and reaches VPQ $8\times$26 and STQ $8\times$27. PMT uses $8\times$28 GFLOPs, runs at $8\times$29 FPS, and reaches VPQ $8\times$30 and STQ $8\times$31 (Cavagnero et al., 26 Mar 2026). The reported interpretation is that PMT runs approximately $8\times$32 faster at a small VPQ/STQ gap of $8\times$33.

On VSPW val, VidEoMT with DINOv2 and a finetuned encoder uses $8\times$34 GFLOPs, runs at $8\times$35 FPS, and obtains mVC $8\times$36 and mIoU $8\times$37. PMT with DINOv3 and a frozen encoder uses $8\times$38 GFLOPs, runs at $8\times$39 FPS, and obtains mVC $8\times$40 and mIoU $8\times$41 (Cavagnero et al., 26 Mar 2026). The paper states that PMT sets a new state-of-the-art mIoU of $8\times$42 with a frozen encoder.

Task	Comparator	PMT
YouTube-VIS 2019/2021	AP19 $8\times$43, $8\times$44 FPS	AP19 $8\times$45, $8\times$46 FPS
VIPSeg val	VPQ $8\times$47, STQ $8\times$48, $8\times$49 FPS	VPQ $8\times$50, STQ $8\times$51, $8\times$52 FPS
VSPW val	mIoU $8\times$53, $8\times$54 FPS	mIoU $8\times$55, $8\times$56 FPS

These video results are particularly relevant to the frequent assumption that frozen-encoder systems are structurally disadvantaged on temporally demanding tasks. PMT does not uniformly dominate every metric-speed pair—for example, VIPSeg shows a small VPQ/STQ gap—but it consistently demonstrates that a frozen encoder can remain competitive across video instance, panoptic, and semantic segmentation when paired with an efficient decoder (Cavagnero et al., 26 Mar 2026).

7. Multi-Task Sharing, Limitations, and Prospective Extensions

Because PMT leaves the encoder untouched, the same frozen VFM can serve multiple tasks with different lightweight PMDs loaded on top, including panoptic, instance, semantic, and video segmentation (Cavagnero et al., 26 Mar 2026). The paper states that this drastically reduces deployment complexity and memory footprint when varied output spaces or label definitions must be supported. In operational terms, the shareable unit is the encoder representation, while task specialization is isolated in the decoder.

The principal limitations are also explicitly identified. PMD relies on sufficiently rich frozen features, and accuracy degrades when smaller encoders or weaker pre-training are used, with ImageNet-1K given as an example (Cavagnero et al., 26 Mar 2026). Spatial detail missing from final encoder layers is only partially recovered via lateral connections, suggesting that more advanced multi-scale fusion or lightweight adapters may close remaining gaps. PMT also addresses per-task decoders separately; the paper notes that unified multi-task decoders or prompt-based extensions could further improve parameter sharing (Cavagnero et al., 26 Mar 2026).

A common misunderstanding is that PMT is simply a frozen reimplementation of encoder-only segmentation. The architecture differs in a substantive way: all task-specific query processing is moved out of the ViT and into a dedicated decoder, while the decoder jointly processes queries and patches with a plain Transformer stack. Conversely, PMT should not be construed as eliminating all trade-offs. The VIPSeg results and the stated reliance on strong frozen features indicate that shareability and speed are obtained within a regime where encoder quality remains a determining factor (Cavagnero et al., 26 Mar 2026).

In that sense, PMT occupies a specific point in the design landscape of VFM-based segmentation: it treats the frozen encoder as a stable, reusable feature interface and shifts the burden of specialization into a lightweight PMD. This suggests a broader architectural principle for multi-task VFM deployment: when frozen representations are sufficiently expressive, decoder-centric adaptation may offer a favorable balance between throughput, modularity, and competitive segmentation accuracy.