---
title: Plain Mask Transformer (PMT)
url: https://www.emergentmind.com/topics/plain-mask-transformer-pmt
type: topic
---

# Plain Mask Transformer (PMT)

Plain Mask Transformer (PMT) is a segmentation architecture designed to preserve the simplicity and low latency of encoder-only designs while keeping a Vision Foundation Model (VFM) encoder frozen and shareable across tasks. Introduced in "PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders" [2603.25398], PMT targets both image and video segmentation by placing a lightweight Transformer decoder, the Plain Mask Decoder (PMD), on top of frozen Vision Transformer (ViT) features. The model is positioned against recent VFM-based encoder-only methods such as EoMT and VidEoMT, which inject learnable queries into the ViT’s self-attention and finetune the entire encoder; PMT relocates task-specific query processing into the decoder and thereby restores multi-task encoder sharing [2603.25398].

## 1. Conceptual Positioning

PMT is motivated by the deployment setting in which modern large-scale VFMs such as DINOv3 produce patch-token representations rich enough to support multiple downstream tasks from a single frozen backbone. In that setting, encoder finetuning undermines one of the principal practical advantages of VFMs: a shared encoder representation that can be reused across panoptic, instance, semantic, and video segmentation tasks without task-specific backbone updates [2603.25398].

Within this framing, PMT can be understood as a reconciliation of two previously competing properties. On one side are encoder-only segmentation systems such as EoMT and VidEoMT, which offer architectural simplicity and speed but require encoder finetuning. On the other side are frozen-encoder systems that preserve backbone shareability but often incur additional complexity or latency. PMT addresses this trade-off by introducing the PMD, a fast Transformer-based segmentation decoder that operates on frozen VFM features while maintaining the generality of the encoder-only framework [2603.25398].

A common misconception is that retaining a frozen encoder necessarily entails a decisive accuracy penalty relative to finetuned alternatives. The PMT results do not support such a blanket conclusion: on standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to $\sim 3\times$ faster, and on video segmentation it performs on par with fully finetuned methods while being up to $8\times$ faster than state-of-the-art frozen-encoder models [2603.25398]. This suggests that, given sufficiently strong frozen features, a substantial portion of task adaptation can be moved into a lightweight decoder without forfeiting the latency profile associated with encoder-only formulations.

## 2. Architectural Composition

PMT consists of two principal components: a frozen ViT encoder pre-trained at scale, and the PMD, a small stack of Transformer layers that jointly processes a fixed set of learnable object queries and multi-depth encoder features [2603.25398].

The processing pipeline is specified as follows. An input image $I \in \mathbb{R}^{3 \times H \times W}$ is passed through a frozen ViT encoder with $L$ layers. Patch tokens $X^l \in \mathbb{R}^{D \times N}$ are extracted at several depths $l_j$, including the final layer $L$. For each selected depth, PMT applies final LayerNorm, then a trainable BatchNorm and a two-layer MLP; the resulting feature maps are summed across depths to obtain multi-scale patch tokens $X^{\text{frozen}} \in \mathbb{R}^{D \times N}$. A fixed set of $K$ learnable queries $Q^{\text{lrn}} = \{q_i^{\text{lrn}} \in \mathbb{R}^D\}_{i=1}^K$ is then concatenated with the patch tokens to form a sequence of length $K+N$, which is processed by $L_d$ Transformer layers in the PMD, typically with $L_d = 6$ [2603.25398].

The decoder produces refined queries $\{\hat{q}_i\} \in \mathbb{R}^{D}$ and refined patch tokens $\hat{X} \in \mathbb{R}^{D \times N}$. Prediction is split into a classification branch and a mask branch. Class logits are defined by
$$
c_i = W_c \hat{q}_i + b_c \in \mathbb{R}^C,
$$
while mask logits are obtained via a dot-product between a small MLP applied to $\hat{q}_i$ and the upsampled patch tokens $X^{\text{up}}$:
$$
M_i \in \mathbb{R}^{(H/4)\times(W/4)}.
$$
In the full formulation, $X^{\text{up}} \in \mathbb{R}^{D \times (H/4) \times (W/4)}$ is $\hat{X}$ reshaped and bilinearly upsampled, and the dot-product is taken between the per-query MLP output of dimension $D$ and each spatial feature vector in $X^{\text{up}}$ [2603.25398].

The design is notable for preserving a plain sequence-processing interface: queries and patches are handled jointly, and the same decoder principle is stated to apply seamlessly to both image and video segmentation. A plausible implication is that PMT inherits some of the implementation regularity of encoder-only systems while relocating all task-specific adaptation into an independently trainable module.

## 3. Plain Mask Decoder Mechanics

The PMD mirrors the architecture of the last $L_2$ layers of EoMT’s finetuned encoder but operates on top of frozen features and is trained from scratch [2603.25398]. Each decoder block $l$ implements the residual-preconditioned Transformer updates
$$
Z^l = X^l + \text{MHSA}(\text{LN}(X^l)),
$$
$$
X^{l+1} = Z^l + \text{FFN}(\text{LN}(Z^l)).
$$
Here, $\text{LN}$ denotes LayerNorm and $\text{MHSA}$ denotes multi-head self-attention.

For an input sequence $X \in \mathbb{R}^{(K+N)\times D}$ at layer $l$, the decoder computes
$$
Q = XW_Q,\quad K = XW_K,\quad V = XW_V,
$$
followed by scaled dot-product attention,
$$
\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V.
$$
All tokens—queries and patches—attend to each other. The paper explicitly identifies this as realizing query self-attention and cross-attention with patches in one operation [2603.25398]. Rotary position embeddings (RoPE) are applied on patch tokens, while queries receive no positional encoding.

A distinctive training-time mechanism is masked attention. At each layer $l$, an intermediate mask $M_i^{(l)}$ is predicted per query, and the attention weights from query positions to patch positions are masked so that attention focuses only on regions where $M_i^{(l)} = 1$. This masking is gradually phased out by a mask-annealing schedule, such that inference uses standard unmasked MHSA, which is stated to be compatible with FlashAttention [2603.25398]. This arrangement couples localized training guidance with inference-time efficiency.

The ablation evidence underscores the functional importance of the decoder itself. On COCO panoptic segmentation, stepwise removal from ViT-Adapter+Mask2Former shows that removing the Transformer decoder in an EoMT-style frozen setting causes PQ to collapse to $6.8$ at $162$ FPS, whereas building up from frozen EoMT with a Plain PMD yields PQ $53.7$ at $145$ FPS; adding lateral connections yields PQ $55.9$ at $142$ FPS; adding RoPE yields PQ $56.1$ at $141$ FPS [2603.25398]. These measurements indicate that PMD is not a marginal accessory but the core mechanism enabling segmentation quality with frozen features.

## 4. Training and Inference Regimen

The encoder remains frozen throughout training; only PMD parameters are updated, including Transformer layers, MLPs in the prediction heads, lateral adapters, and BatchNorms [2603.25398]. This training boundary is central to PMT’s intended role as a shareable-head architecture on top of a common VFM.

The supervision setup comprises cross-entropy for classification between predicted $c_i$ and ground-truth class, together with binary cross-entropy plus Dice loss for mask prediction between $M_i$ and the ground-truth mask [2603.25398]. Optimization uses AdamW. For image models, the reported configuration is batch size $16$, learning rate $2 \times 10^{-4}$, cosine annealing, $12$ epochs on COCO, $16$ epochs on ADE20K, and a $6\,000$-step linear warmup. For video models, the configuration is batch size $8$ with a $5$-frame window, learning rate $1 \times 10^{-4}$, polynomial decay with power $0.9$, a training schedule following CAVIS, and a $6\,000$-step warmup. Data augmentations follow EoMT/VidEoMT protocols, including random flip and scale jitter [2603.25398].

Inference speed is measured on NVIDIA H100 with FlashAttention-2 and `torch.compile`, using batch size $1$ and mixed precision [2603.25398]. For images, PMT-L with ViT-L runs at $141$ FPS versus ViT-Adapter+Mask2Former at $48$ FPS, corresponding to approximately $3\times$ higher throughput. For video, PMT-L runs at $112$–$124$ FPS versus frozen-encoder baselines at $13$–$15$ FPS, yielding up to approximately $8\times$ higher throughput [2603.25398].

These choices clarify that PMT’s efficiency claims are not limited to algorithmic FLOP reductions. They are tied to an inference path that avoids training-time masking, uses standard unmasked MHSA at test time, and is explicitly aligned with high-performance attention kernels.

## 5. Empirical Results on Image Segmentation

On COCO `val2017`, PMT is compared against ViT-Adapter + Mask2Former in the frozen-encoder regime. At resolution $640^2$, ViT-Adapter + Mask2Former with DINOv3 and a frozen pyramid uses $340$M parameters, $804$ GFLOPs, runs at $48$ FPS, and attains PQ $56.4$ and AP $45.9$. PMT with DINOv3 and a frozen pyramid uses $357$M parameters, $767$ GFLOPs, runs at $141$ FPS, and attains PQ $56.1$ and AP $45.4$ [2603.25398]. At $1280^2$, ViT-Adapter + M2F reaches PQ $58.1$ at $15$ FPS, while PMT reaches PQ $58.1$ at $29$ FPS, i.e., approximately $2\times$ faster [2603.25398].

On ADE20K `val`, ViT-Adapter + M2F with DINOv3 uses $340$M parameters and $879$ GFLOPs, runs at $40$ FPS, and obtains mIoU $58.7$. PMT uses $357$M parameters and $823$ GFLOPs, runs at $128$ FPS, and obtains mIoU $58.5$ [2603.25398]. The paper summarizes this as PMT being more than $3\times$ faster with less than $0.3$ mIoU drop.

| Benchmark | Baseline result | PMT result |
|---|---:|---:|
| COCO `val2017` @ $640^2$ | PQ $56.4$, AP $45.9$, $48$ FPS | PQ $56.1$, AP $45.4$, $141$ FPS |
| COCO `val2017` @ $1280^2$ | PQ $58.1$, $15$ FPS | PQ $58.1$, $29$ FPS |
| ADE20K `val` | mIoU $58.7$, $40$ FPS | mIoU $58.5$, $128$ FPS |

The image results characterize PMT as a frozen-encoder model that remains very close to the strongest reported frozen-encoder baselines in quality while materially shifting the speed-accuracy operating point [2603.25398]. A plausible implication is that, for applications constrained by throughput or latency rather than by the last fraction of a point in PQ or mIoU, PMT changes the practical baseline for frozen-VFM segmentation.

## 6. Empirical Results on Video Segmentation

On YouTube-VIS 2019/2021 at $640^2$, CAVIS with DINOv3 and a frozen encoder uses $838$ GFLOPs, runs at $15$ FPS, and attains AP19 $68.5$, AP$_{75}$ $75.8$, and AR$_{10}$ $73.5$. PMT with DINOv3 and a frozen encoder uses $617$ GFLOPs, runs at $113$ FPS, and attains AP19 $69.2$, AP$_{75}$ $76.5$, and AR$_{10}$ $74.6$ [2603.25398]. The reported summary is that PMT is approximately $7.5\times$ faster while slightly improving AP.

On VIPSeg `val`, CAVIS with DINOv3 and a frozen encoder uses $2612$ GFLOPs, runs at $9$ FPS, and reaches VPQ $56.8$ and STQ $51.2$. PMT uses $2037$ GFLOPs, runs at $58$ FPS, and reaches VPQ $55.5$ and STQ $49.2$ [2603.25398]. The reported interpretation is that PMT runs approximately $6\times$ faster at a small VPQ/STQ gap of $1.3/2.0$.

On VSPW `val`, VidEoMT with DINOv2 and a finetuned encoder uses $1909$ GFLOPs, runs at $73$ FPS, and obtains mVC $95.0$ and mIoU $64.9$. PMT with DINOv3 and a frozen encoder uses $2049$ GFLOPs, runs at $57$ FPS, and obtains mVC $94.9$ and mIoU $65.7$ [2603.25398]. The paper states that PMT sets a new state-of-the-art mIoU of $65.7$ with a frozen encoder.

| Task | Comparator | PMT |
|---|---:|---:|
| YouTube-VIS 2019/2021 | AP19 $68.5$, $15$ FPS | AP19 $69.2$, $113$ FPS |
| VIPSeg `val` | VPQ $56.8$, STQ $51.2$, $9$ FPS | VPQ $55.5$, STQ $49.2$, $58$ FPS |
| VSPW `val` | mIoU $64.9$, $73$ FPS | mIoU $65.7$, $57$ FPS |

These video results are particularly relevant to the frequent assumption that frozen-encoder systems are structurally disadvantaged on temporally demanding tasks. PMT does not uniformly dominate every metric-speed pair—for example, VIPSeg shows a small VPQ/STQ gap—but it consistently demonstrates that a frozen encoder can remain competitive across video instance, panoptic, and semantic segmentation when paired with an efficient decoder [2603.25398].

## 7. Multi-Task Sharing, Limitations, and Prospective Extensions

Because PMT leaves the encoder untouched, the same frozen VFM can serve multiple tasks with different lightweight PMDs loaded on top, including panoptic, instance, semantic, and video segmentation [2603.25398]. The paper states that this drastically reduces deployment complexity and memory footprint when varied output spaces or label definitions must be supported. In operational terms, the shareable unit is the encoder representation, while task specialization is isolated in the decoder.

The principal limitations are also explicitly identified. PMD relies on sufficiently rich frozen features, and accuracy degrades when smaller encoders or weaker pre-training are used, with ImageNet-1K given as an example [2603.25398]. Spatial detail missing from final encoder layers is only partially recovered via lateral connections, suggesting that more advanced multi-scale fusion or lightweight adapters may close remaining gaps. PMT also addresses per-task decoders separately; the paper notes that unified multi-task decoders or prompt-based extensions could further improve parameter sharing [2603.25398].

A common misunderstanding is that PMT is simply a frozen reimplementation of encoder-only segmentation. The architecture differs in a substantive way: all task-specific query processing is moved out of the ViT and into a dedicated decoder, while the decoder jointly processes queries and patches with a plain Transformer stack. Conversely, PMT should not be construed as eliminating all trade-offs. The VIPSeg results and the stated reliance on strong frozen features indicate that shareability and speed are obtained within a regime where encoder quality remains a determining factor [2603.25398].

In that sense, PMT occupies a specific point in the design landscape of VFM-based segmentation: it treats the frozen encoder as a stable, reusable feature interface and shifts the burden of specialization into a lightweight PMD. This suggests a broader architectural principle for multi-task VFM deployment: when frozen representations are sufficiently expressive, decoder-centric adaptation may offer a favorable balance between throughput, modularity, and competitive segmentation accuracy.

Source: https://www.emergentmind.com/topics/plain-mask-transformer-pmt