Plain Mask Decoder (PMD) Overview

• Plain Mask Decoder (PMD) is a lightweight Transformer-based decoder that processes pre-computed frozen VFM features to generate per-query class logits and dense masks.
• It uses a lateral-connection feature aggregator, rotary positional encoding, and a stack of Transformer layers with minimal mask and classification heads to maintain simplicity and low latency.
• PMD enables both image and video segmentation by preserving frozen encoder representations, achieving 3–8× speed improvements while maintaining competitive accuracy across tasks.

Plain Mask Decoder (PMD) is a small Transformer-based decoder introduced in the Plain Mask Transformer (PMT) for image and video segmentation with frozen Vision Foundation Model (VFM) encoders. It is designed to operate on pre-computed, frozen VFM features, taking a set of learnable object queries and patch tokens extracted from multiple layers of a frozen ViT encoder, and producing per-query class logits and dense masks. In PMT, PMD is the mechanism that preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable across tasks and datasets (Cavagnero et al., 26 Mar 2026).

1. Position within frozen-VFM segmentation

PMD is defined in contrast to recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, which achieve competitive accuracy with remarkably low latency but require finetuning the encoder. PMD is proposed to reconcile encoder-only simplicity and speed with frozen VFM features. The resulting PMT keeps the ViT encoder fully frozen, and PMD plus the two small MLP heads are the only trainable components (Cavagnero et al., 26 Mar 2026).

Within PMT, PMD sits on top of a pre-computed, frozen Vision Foundation Model encoder, for example DINOv3 ViT-L. Its inputs are a set of $K$ learnable object queries $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$ and patch tokens extracted from multiple layers of the frozen ViT encoder, $X^l \in \mathbb{R}^{N \times D}$ for $l \in \{l_1,\dots,l_m\}$, where $N=(H/p)\cdot(W/p)$. Its outputs are, for each query, class logits $c_i \in \mathbb{R}^C$ and a dense mask $M_i \in \mathbb{R}^{H/4 \times W/4}$ (Cavagnero et al., 26 Mar 2026).

This placement is operationally significant because, for image segmentation, PMT runs one forward pass through the frozen ViT, gathers multi-depth features, and decodes them with PMD to obtain per-query class and mask predictions. For online video segmentation, frames are processed independently through the frozen encoder, with temporal context carried solely in the query vectors. A plausible implication is that the frozen encoder can remain a common substrate for multiple downstream segmentation workloads without task-specific encoder updates.

2. Module composition and decoding pipeline

PMD consists of four main modules: a lateral-connection feature aggregator, positional encoding, a stack of $L_d$ Transformer layers, and a small mask-and-classification head (Cavagnero et al., 26 Mar 2026).

The pipeline is specified as follows. An input image is passed through a frozen ViT encoder, and patch tokens $X^l$ are extracted from several depths $l$. These features are processed by applying the final LayerNorm $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$0 together with trainable BatchNorm and a 2-layer MLP, after which the contributions from different depths are summed to produce $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$1. Rotary positional encoding is applied to $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$2, the learnable queries $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$3 are concatenated, and this forms the initial decoder sequence $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$4. After $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$5 Transformer layers, the final sequence is split into updated patch tokens $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$6 and updated query tokens $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$7, which are then passed to the mask and class head to produce $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$8 (Cavagnero et al., 26 Mar 2026).

For video, the only change is in the queries. Instead of always feeding the static $Q^{\mathrm{lrn}} \in \mathbb{R}^{K \times D}$9, PMD uses

$X^l \in \mathbb{R}^{N \times D}$0

and then proceeds identically on each frame (Cavagnero et al., 26 Mar 2026).

This decoder structure is notable for what it omits. The paper attributes part of the low-latency behavior to the absence of any large pixel decoder, large convolutions, or multi-scale fusions in the mask head. This suggests that PMD is intended as a minimal but task-effective decoding layer over frozen VFM features rather than as a replacement for the encoder itself.

3. Mathematical formulation

The core Transformer computation follows standard multi-head self-attention over a mixed sequence of spatial tokens and query tokens. Let $X^l \in \mathbb{R}^{N \times D}$1 be a generic sequence of $X^l \in \mathbb{R}^{N \times D}$2 tokens, with $X^l \in \mathbb{R}^{N \times D}$3. The query, key, and value matrices are formed as

$X^l \in \mathbb{R}^{N \times D}$4

with $X^l \in \mathbb{R}^{N \times D}$5. These are split into $X^l \in \mathbb{R}^{N \times D}$6 heads of size $X^l \in \mathbb{R}^{N \times D}$7, and for head $X^l \in \mathbb{R}^{N \times D}$8 the per-head attention is

$X^l \in \mathbb{R}^{N \times D}$9

Concatenation and output projection give

$l \in \{l_1,\dots,l_m\}$0

where $l \in \{l_1,\dots,l_m\}$1. Each Transformer layer then applies residual attention and feed-forward updates:

$l \in \{l_1,\dots,l_m\}$2

$l \in \{l_1,\dots,l_m\}$3

with

$l \in \{l_1,\dots,l_m\}$4

where $l \in \{l_1,\dots,l_m\}$5, $l \in \{l_1,\dots,l_m\}$6, and $l \in \{l_1,\dots,l_m\}$7 is the MLP expansion ratio (Cavagnero et al., 26 Mar 2026).

The lateral-connection aggregation stage extracts patch tokens from $l \in \{l_1,\dots,l_m\}$8 evenly spaced encoder layers, including the final layer $l \in \{l_1,\dots,l_m\}$9. Each $N=(H/p)\cdot(W/p)$0 is normalized and projected through

$N=(H/p)\cdot(W/p)$1

$N=(H/p)\cdot(W/p)$2

which the paper describes as “trainable BN” on top of frozen LayerNorm, and

$N=(H/p)\cdot(W/p)$3

The aggregated token representation is then

$N=(H/p)\cdot(W/p)$4

Following DINOv3, Rotary Positional Encoding (RoPE) is applied to $N=(H/p)\cdot(W/p)$5 pairs in each MHSA so that relative 2D positions are encoded without per-token learnable embeddings. Concretely, each head’s query vector for position $N=(H/p)\cdot(W/p)$6 is multiplied by a fixed orthonormal rotation $N=(H/p)\cdot(W/p)$7, and likewise for the key (Cavagnero et al., 26 Mar 2026).

After the decoder stack, the sequence is split into $N=(H/p)\cdot(W/p)$8 and $N=(H/p)\cdot(W/p)$9. The class logits for query $c_i \in \mathbb{R}^C$0 are

$c_i \in \mathbb{R}^C$1

with $c_i \in \mathbb{R}^C$2. For mask prediction,

$c_i \in \mathbb{R}^C$3

and mask logits are computed as

$c_i \in \mathbb{R}^C$4

for spatial location $c_i \in \mathbb{R}^C$5 in an upsampled version $c_i \in \mathbb{R}^C$6 of $c_i \in \mathbb{R}^C$7. After reshaping $c_i \in \mathbb{R}^C$8 to $c_i \in \mathbb{R}^C$9, the foreground probability at each pixel is

$M_i \in \mathbb{R}^{H/4 \times W/4}$0

Training uses classification cross-entropy on $M_i \in \mathbb{R}^{H/4 \times W/4}$1 matched to ground-truth objects, together with binary cross-entropy and Dice loss on each mask $M_i \in \mathbb{R}^{H/4 \times W/4}$2 versus the matched ground-truth mask (Cavagnero et al., 26 Mar 2026).

4. Design choices and implementation parameters

The decoder design is specified with a compact set of hyperparameters. The number of decoder layers is $M_i \in \mathbb{R}^{H/4 \times W/4}$3, and the ablation shows saturation by 6. The hidden dimension $M_i \in \mathbb{R}^{H/4 \times W/4}$4 matches the encoder, for example $M_i \in \mathbb{R}^{H/4 \times W/4}$5 for ViT-L. The number of heads $M_i \in \mathbb{R}^{H/4 \times W/4}$6 also matches the encoder, for example $M_i \in \mathbb{R}^{H/4 \times W/4}$7, and the FFN expansion ratio $M_i \in \mathbb{R}^{H/4 \times W/4}$8 matches the encoder design, for example $M_i \in \mathbb{R}^{H/4 \times W/4}$9. Lateral connections are taken from $L_d$0 depths, including the final layer, and RoPE is used in all decoder layers (Cavagnero et al., 26 Mar 2026).

Training uses mask annealing to gradually remove masked-attention. The optimizer is AdamW, with learning rate $L_d$1 for image tasks and $L_d$2 for video tasks, together with mixed precision and linear warmup. The batch size is 16 for image tasks and 8 for video tasks, with standard augmentations (Cavagnero et al., 26 Mar 2026).

The paper also gives an explicit efficiency rationale. By matching the encoder’s size and attention mechanics, PMD recovers almost all of EoMT’s performance in only $L_d$3 layers, instead of re-finetuning the entire ViT. Lateral connections bring multi-scale cues such as edges and textures from intermediate layers without any large pixel decoder. RoPE makes spatial relations explicit, so no learned positional embeddings are needed. The mask head is a single small dot-product MLP, avoiding large convolutions or multi-scale fusions. Mask annealing removes all custom attention masks at inference, enabling fused QKV projections and FlashAttention for maximal speed (Cavagnero et al., 26 Mar 2026).

A common misconception in this design space is that competitive segmentation accuracy with VFM features necessarily requires either encoder finetuning or substantial adapter machinery. PMD is presented specifically as an alternative to that assumption: the encoder remains untouched, while the decoder is tailored to the encoder’s representational and attention structure.

5. Image and video operation

In image segmentation, PMD processes the frozen encoder features from a single frame and produces per-query class and mask outputs. In video segmentation, the encoder still processes frames independently, and temporal context is carried solely in the query vectors through the recurrence

$L_d$4

The design therefore keeps the video variant close to the image variant, with the only change in the queries (Cavagnero et al., 26 Mar 2026).

This video mechanism is important because it constrains temporal modeling to the decoder state rather than the encoder representation. The paper states that the design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. A plausible implication is that the same frozen VFM can support multiple segmentation modalities while the temporal specialization is isolated to a lightweight recurrent query path.

Because the encoder is never touched, a single frozen VFM can be shared across any number of segmentation tasks or datasets. In deployment terms, this is the central systems-level motivation for PMD: it preserves the practical attractiveness of frozen VFM sharing while retaining the low-latency profile associated with encoder-only segmentation architectures (Cavagnero et al., 26 Mar 2026).

6. Reported performance and efficiency

The reported measurements are given on an NVIDIA H100 with FlashAttention-2, torch.compile, and batch size 1. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to approximately $L_d$5 faster. For video segmentation, it performs on par with fully finetuned methods while being up to $L_d$6 faster than state-of-the-art frozen-encoder models (Cavagnero et al., 26 Mar 2026).

Benchmark	PMT result	Comparator result
COCO val2017 (640×640)	PMT (DINOv3, ViT-L): 357 M params, 767 GFLOPs, 141 FPS, PQ=56.1	ViT-Adapter + Mask2Former: 340 M params, 804 GFLOPs, 48 FPS, PQ=56.4
ADE20K (512×512)	PMT (DINOv3, ViT-L): 357 M, 823 GFLOPs, 128 FPS, mIoU=58.5	Mask2Former+Adapter: 340 M, 879 GFLOPs, 40 FPS, mIoU=58.7
YouTube-VIS 2019	PMT (DINOv3, ViT-L): 617 M, 113 FPS, AP=69.2, AP\$L_d$7=76.5</td> <td>CAVIS: ≈824 M flops, 15 FPS, AP=68.5, AP\$L_d$8=75.8
VIPSeg	PMT (DINOv3): 2,037 GFLOPs, 58 FPS, VPQ=55.5, STQ=49.2	CAVIS: ∼2,612 GFLOPs, 9 FPS, VPQ=56.8, STQ=51.2
VSPW	PMT (DINOv3): 2,049 GFLOPs, 57 FPS, mIoU=65.7, mVC=94.9	VidEoMT: 1,909 GFLOPs, 71 FPS, mIoU=64.0, mVC=94.4

The paper summarizes these comparisons as follows. On COCO val2017, PMT is approximately $L_d$9 faster at essentially identical PQ, with a difference of $X^l$0. On ADE20K, PMT is $X^l$1 faster at $X^l$2 mIoU. On YouTube-VIS 2019, PMT is more than $X^l$3 faster and slightly more accurate than CAVIS. On VIPSeg, PMT runs more than $X^l$4 faster, with a small drop in VPQ. On VSPW, PMT sets a new state of the art in mIoU while keeping the encoder frozen (Cavagnero et al., 26 Mar 2026).

The paper’s concluding interpretation is that PMD strikes a trade-off in which a few carefully designed Transformer layers plus lightweight heads plug seamlessly on top of a frozen VFM and yield segmentation quality on par with the best finetuned or adapter-based models, yet run $X^l$5–$X^l$6 faster. This suggests that, within the PMT framework, PMD is not merely a decoder component but the key architectural device by which frozen-VFM segmentation is made simultaneously general, efficient, and competitive across image and video settings.