Patch-as-Decodable Token (PaDT)

• PaDT is a paradigm for multimodal LLMs that treats image patches as decodable tokens, unifying text and visual outputs.
• It introduces Visual Reference Tokens (VRTs) derived from patch embeddings, enabling the model to output detection, segmentation, and grounding results in one sequence.
• Empirical results display significant gains in REC, RES, and OVD tasks, affirming PaDT’s robust design and unified multimodal approach.

Patch-as-Decodable Token (PaDT) is a paradigm for multimodal LLMs in which image patches are elevated from input-side latent features to output-side decodable symbols. In its canonical formulation, introduced in "Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs" (Su et al., 2 Oct 2025), PaDT enables an MLLM to autoregressively generate ordinary text tokens and image-conditioned visual tokens in a single sequence, then decode the visual tokens into detection, segmentation, grounding, and grounded captioning outputs. Its central mechanism is the Visual Reference Token (VRT): a token derived from query-image patch embeddings, dynamically inserted into the model’s vocabulary for the current forward pass and interpreted by a lightweight decoder as a sparse, grounded reference to visual content.

1. Definition and motivation

PaDT was proposed to address a specific limitation of conventional MLLMs: they typically accept multimodal input but emit only text tokens as output (Su et al., 2 Oct 2025). This restriction is workable for captioning and visual question answering, but it is structurally awkward for object detection, referring expression comprehension, referring expression segmentation, visual grounding, and grounded captioning. In those settings, prior methods often serialize geometry as text, such as coordinate strings of the form $[x_1,y_1,x_2,y_2]$. The PaDT formulation argues that this is brittle because output format can vary, coordinates are fragmented across multiple text tokens, and text strings do not inherently preserve semantic alignment with image regions (Su et al., 2 Oct 2025).

The PaDT alternative is to let the model directly generate tokens that correspond to image patches from the query image. In the original system, these are VRTs, derived from patch embeddings and interleaved seamlessly with textual outputs. A lightweight decoder then converts groups of generated VRTs into boxes, masks, and scores (Su et al., 2 Oct 2025). This suggests that PaDT is not merely a different output encoding, but a change in the representational status of visual patches: they become referable symbols in the autoregressive output space rather than passive hidden states.

Within this formulation, “unified multi-modal vision tasks” has two precise senses. First, textual and visual outputs inhabit one shared decoding process. Second, multiple tasks can be solved by prompting a single MLLM rather than switching model families. The evaluated tasks in the original PaDT work are referring expression comprehension (REC), referring expression segmentation (RES), open-vocabulary detection (OVD), and referring image captioning (RIC) (Su et al., 2 Oct 2025).

2. Architectural mechanism

The PaDT pipeline extends a standard MLLM with a dynamic visual token mechanism and a lightweight task decoder (Su et al., 2 Oct 2025). Given an image

$I \in \mathbb{R}^{H \times W \times 3},$

a visual encoder $f_v$ partitions it into $N$ non-overlapping patches and produces

$F_v = f_v(I) \in \mathbb{R}^{N \times d_v}.$

A projector $f_p$ then aligns dimensions and downsamples to

$$F_{\text{patch}} = f_p(F_v) \in \mathbb{R}^{N' \times d}.$$

In the Qwen2.5-VL setting used by the paper, nearest-neighbor patch merging in 2D patch space gives $N'=\frac{1}{4}N$ (Su et al., 2 Oct 2025).

VRTs are formed by projecting the aligned patch features into visual reference prototypes:

$$\mathcal{P}_{\text{ref}} = f_{vp}(F_{\text{patch}}) \in \mathbb{R}^{N' \times d}.$$

The module $f_{vp}$ consists of a LayerNorm and a low-rank linear projection (Su et al., 2 Oct 2025). These prototypes are then appended to the text embedding table to create a dynamic embedding table,

$I \in \mathbb{R}^{H \times W \times 3},$0

and the output classifier is expanded in parallel as

$I \in \mathbb{R}^{H \times W \times 3},$1

This is the core architectural move: the current image contributes additional, image-specific symbols to both the input embedding space and the output vocabulary (Su et al., 2 Oct 2025).

During decoding, textual and visual tokens are emitted in one autoregressive stream. The paper’s templates make the mechanism explicit. In REC and RES, an assistant response can include phrases such as “The ‘{question}’ refers to <VRT_0><VRT_1><VRT_2><VRT_3><VRT_4> in this image.” In OVD, category text and VRT groups are interleaved. In RIC, object mentions are followed by corresponding VRT sequences (Su et al., 2 Oct 2025). VRT groups are then extracted from the final LLM hidden states and passed to the PaDT decoder.

The decoder is a lightweight stack of three two-way attention blocks. For each VRT group, it injects three learnable tokens—a bounding box token, a mask token, and a score token—and outputs a box, a segmentation mask, and a confidence score (Su et al., 2 Oct 2025). The final geometry is therefore not generated token by token; rather, VRTs are generated autoregressively, while the box and mask are decoded non-autoregressively from the hidden states of those generated VRTs.

3. Training objective and task unification

Training PaDT requires supervision over both sequence generation and structured visual prediction (Su et al., 2 Oct 2025). The standard autoregressive loss is defined over the expanded token space:

$I \in \mathbb{R}^{H \times W \times 3},$2

However, the paper introduces a robust per-token cross-entropy because many foreground VRTs may be valid references for the same target. A foreground mask $I \in \mathbb{R}^{H \times W \times 3},$3 identifies unselected VRTs, and logits are modified as

$I \in \mathbb{R}^{H \times W \times 3},$4

The robust objective is then

$I \in \mathbb{R}^{H \times W \times 3},$5

Unselected foreground VRTs are thus removed from softmax competition rather than treated as negatives (Su et al., 2 Oct 2025).

A second training mechanism is random VRT selection. The model samples $I \in \mathbb{R}^{H \times W \times 3},$6 visual reference tokens per target. If segmentation masks are available, VRTs are sampled from the foreground mask; otherwise, they are sampled from inside the bounding box (Su et al., 2 Oct 2025). The stated motivation is increased supervision diversity and improved generalization. Empirically, this is a decisive design choice: using all foreground VRTs leads to severe degradation (Su et al., 2 Oct 2025).

Task-specific decoder losses are added on top of sequence supervision. The box loss is

$I \in \mathbb{R}^{H \times W \times 3},$7

the mask loss is

$I \in \mathbb{R}^{H \times W \times 3},$8

and the score loss is

$I \in \mathbb{R}^{H \times W \times 3},$9

The final objective is

$f_v$0

This loss structure is what makes PaDT a genuinely unified multimodal output system rather than a text-only MLLM with a post hoc detector (Su et al., 2 Oct 2025).

4. Empirical performance

In the original experiments, PaDT is built on Qwen2.5-VL in 3B and 7B variants and evaluated on REC, RES, OVD, and RIC (Su et al., 2 Oct 2025). The strongest multi-task version, PaDT Pro, reports REC overall scores of 93.6 for 3B and 94.5 for 7B. For RES, PaDT reports overall scores of 73.4 for 3B and 75.8 for 7B, while PaDT Pro reports 79.4 for 3B and 84.1 for 7B (Su et al., 2 Oct 2025).

The OVD results are particularly large relative to prior MLLM baselines. Qwen2.5-VL 3B reports 13.7 AP, VLM-R1 3B reports 19.2 AP, PaDT 3B reports 34.0 AP, and PaDT Pro 3B reports 38.2 AP. The 7B variants report 36.5 AP for PaDT and 39.0 AP for PaDT Pro (Su et al., 2 Oct 2025). In RIC, PaDT 3B reaches CIDEr 1.450 with GP 81.6 and GR 45.4, while PaDT Pro 7B reports GP 82.3 and GR 45.8 (Su et al., 2 Oct 2025).

Ablations isolate the contribution of the patch-decodable interface. A supervised Qwen2.5-VL baseline without VRTs reaches 88.7 on REC; introducing VRTs, robust CE, and random selection yields 93.2 on REC and makes RES possible with strong performance (Su et al., 2 Oct 2025). Adding the projection module $f_v$1 improves REC from 91.1 to 92.0 before robust CE is added (Su et al., 2 Oct 2025). The patch-count ablation shows that 5 patches is best, while “ALL” causes catastrophic drop. The reported REC/RES values for all-VRT supervision are 76.5/69.5 without robust CE and 49.1/19.8 with robust CE, versus 93.2/76.1 with random selected VRTs and robust CE (Su et al., 2 Oct 2025).

Further ablations show that removing the dynamic embedding module or any of $f_v$2, $f_v$3, or $f_v$4 degrades performance, and Objects365 pretraining improves COCO OVD from 34.0 AP to 36.5 AP after finetuning (Su et al., 2 Oct 2025). The paper also reports compatibility with SAM2-L refinement: on RefCOCOg val, the best combination of prompts reaches 76.3 versus 70.5 without refinement (Su et al., 2 Oct 2025).

5. Relation to adjacent patch-level formulations

PaDT has also become a useful lens for interpreting adjacent work that treats patches as more than fixed input units. These formulations are closely related but not equivalent.

The nearest visual autoregressive analogue is DPAR, which dynamically aggregates contiguous VQ tokens into adaptive patches and runs the expensive global transformer over the shorter patch sequence (Srivastava et al., 26 Dec 2025). In PaDT terms, DPAR is described as a “dynamic PaDT-style formulation,” but it is not canonical PaDT because it does not introduce a discrete patch vocabulary. Instead, grouped original tokens serve as the units of autoregressive computation, and a lightweight decoder maps patch states back to the original token vocabulary. Its likelihood remains token-supervised, boundaries are externally determined by entropy thresholding, and geometry is restricted to within-row contiguous spans (Srivastava et al., 26 Dec 2025).

NPP, or Next Patch Prediction, is another precursor-like formulation in autoregressive image generation. It groups non-overlapping image tokens into patch embeddings via averaging,

$f_v$5

and supervises one patch-level prediction against the constituent token labels (Pang et al., 2024). NPP therefore moves the prediction granularity from single tokens to patches, but the patch is not an explicit decodable symbol with its own index. The paper itself is best characterized as “patch-as-prediction-target” rather than full patch-as-decodable-token (Pang et al., 2024).

Outside vision generation, "From Characters to Tokens: Dynamic Grouping with Hierarchical BPE" offers an explicitly decodable textual analogue. It treats each first-stage BPE token as a patch, appends an explicit end-of-patch symbol, and performs a second BPE compression stage so that each patch remains short enough for local encoding and decoding (Dolga et al., 17 Oct 2025). This yields self-delimiting, lossless, variable-length patches without an auxiliary boundary model, which is highly PaDT-aligned even though each patch is represented as a short sequence rather than a single atomic token (Dolga et al., 17 Oct 2025).

"Scratchpad Patching" addresses a different issue: patch lag in byte-level LLMs. Standard patching forces most byte predictions inside a patch to rely on stale previous-patch context; Scratchpad Patching inserts transient within-patch states to refresh patch-level context without increasing persistent KV-cache size (Zheng et al., 10 May 2026). The paper does not present patches as standalone decodable tokens, but it identifies causal freshness as a structural precondition for any patch-level predictive unit (Zheng et al., 10 May 2026).

Two additional lines of work bear directly on PaDT’s representational assumptions. "Revisiting [CLS] and Patch Token Interaction in Vision Transformers" shows that shared processing creates friction between global-purpose [[CLS](https://www.emergentmind.com/topics/chunk-level-sparsity-cls)] tokens and local patch tokens, and that token-type-aware normalization and early QKV specialization substantially improve patch representation quality for dense prediction (Marouani et al., 9 Feb 2026). "Differentiable Hierarchical Visual Tokenization" replaces fixed square patches with content-adaptive connected regions selected from a hierarchical superpixel tree; these tokens are grounded in interpretable supports and even support out-of-the-box raster-to-vector conversion, though they are not explicit parametric decodable tokens in the strict PaDT sense (Aasan et al., 4 Nov 2025).

6. Decodability, limits, and broader implications

The PaDT literature distinguishes, implicitly and sometimes explicitly, between semantic decodability and structural decodability. In the original PaDT model, VRTs are semantically grounded output symbols because they refer to actual query-image patches and are decoded into boxes and masks (Su et al., 2 Oct 2025). In adjacent work, the degree of decodability varies. DPAR uses patches as state-compression units rather than atomic patch symbols (Srivastava et al., 26 Dec 2025). NPP supervises patch-level prediction without a patch vocabulary (Pang et al., 2024). Scratchpad Patching improves causal usability of patches without making them public outputs (Zheng et al., 10 May 2026).

A different perspective comes from the black-box attack paper "Steal the Patch Size: Adversarially Manipulate Vision-LLMs" (Hu et al., 30 Jun 2026). That work shows that patch size, grid alignment, and parts of the preprocessing pipeline are behaviorally identifiable from end-task outputs alone by exploiting periodic accuracy collapses when synthetic grid boundaries align with hidden patch boundaries. The paper does not establish semantic decodability of individual patch contents; it establishes structural decodability of tokenization geometry (Hu et al., 30 Jun 2026). For PaDT, this is significant because it shows that patches are not merely internal engineering details: their granularity and placement can leave measurable external signatures.

The limitations of PaDT are correspondingly concrete. In the original system, performance still depends on the visual backbone’s patch resolution, the method requires a decoder to recover boxes and masks from sparse VRT groups, and results depend strongly on selecting a suitable number of representative patches rather than all foreground patches (Su et al., 2 Oct 2025). Related formulations reveal further constraints: DPAR depends on entropy estimates and restricts patch geometry to within-row spans (Srivastava et al., 26 Dec 2025); dHT’s tokens are interpretable regions, but vectorization is post hoc rather than natively decoded from token parameters (Aasan et al., 4 Nov 2025); Scratchpad Patching shows that larger patches create causal staleness unless refreshed within the patch (Zheng et al., 10 May 2026).

Taken together, these results suggest a broader research trajectory. PaDT in the strict sense is a query-image-specific output-vocabulary expansion in which visual patches become decodable symbols for an MLLM (Su et al., 2 Oct 2025). Surrounding work indicates several adjacent requirements for making such symbols effective: adaptive granularity, decoder-friendly patch embeddings, causal freshness within a patch, and robustness to token-role interference (Srivastava et al., 26 Dec 2025). A plausible implication is that future PaDT-style systems will combine explicit output-side patch tokens with stronger adaptive tokenization and token-type-aware internal processing, while preserving the core PaDT principle that patches can function as first-class, autoregressively generated visual references rather than indirect textual surrogates.