Plug-and-Play Visual Decoder: Modular Vision Extension

• The paper introduces plug-and-play visual decoders as modular attachments that extend existing vision systems without full retraining.
• They utilize architectural paradigms like auxiliary-token stream attachments and retrofitted decoding heads to enable scalable, task-specific improvements.
• Empirical results demonstrate notable gains in segmentation IoU, BD-Rate reduction, and decoding speed, underscoring practical efficiency and adaptability.

A Plug-and-Play Visual Decoder is a modular, attachable visual inference module designed to extend or accelerate existing vision or multimodal systems without end-to-end retraining or invasive architectural modification. Distinct from tightly coupled, task-specific decoders, plug-and-play visual decoders introduce task or efficiency improvements post hoc, leveraging existing latent codes, attention maps, or intermediate features. This approach emphasizes architectural flexibility, extensibility, and often preservation of upstream model generalization—enabling rapid adaptation to new visual tasks, hardware constraints, or rate–distortion requirements.

1. Foundations and Motivation

Plug-and-play visual decoders emerged in response to the inflexibility of task-coupled and monolithic visual inference pipelines. In traditional vision-for-machines or multimodal LLM (MLLM) settings, visual decoding had to be integrated during end-to-end training, resulting in:

• Tight coupling between the codec/representation and the task model
• High retraining costs upon task or model changes
• Limited scalability to heterogeneous or evolving downstream requirements

Plug-and-play decoders address these issues by exposing minimally sufficient, frozen interfaces (e.g., quantized latents, attention maps) that can be interpreted, refined, or remixed by external, lightweight modules. Common motivations include:

• Task scalability: Adding new outputs (e.g., segmentation, depth, recognition) without retraining the baseline codec or MLLM
• Efficient model upgrade: Swapping decoders to optimize for speed or hardware constraints with no latent-space realignment
• Multi-user/multitask deployment: Supporting humans and machines, or varied downstream tasks, over a single compressed stream

Representative examples are found in video coding for machines (PAT-VCM (Jiang et al., 14 Apr 2026)), MLLM segmentation heads (LENS (Liu et al., 19 Oct 2025)), scalable image coding (VVC+M (Harell et al., 2023)), and accelerated generative VAE decoding (Flash-VAED (Zhu et al., 22 Feb 2026)).

2. Architectural Paradigms

Plug-and-play visual decoders fall largely into two architectural paradigms:

Auxiliary-Token Stream Attachments

As in PAT-VCM, the baseline encoder/codec produces a universal, frozen latent stream ($z_0$), while task-specific auxiliary token streams supplement this with minimal, dedicated bitstreams per downstream task. Auxiliary branches come in several forms:

• Visual residual tokens: Provide pixel-space refinements for ROI-centric decoding.
• Prompt/control tokens: Convey compact control signals (e.g., point-based prompts) to steer frozen decoders.
• Semantic tokens: Encode high-level discrete semantics (e.g., class labels) using external VLMs.

At decode time, the core latent is always valid and auxiliary streams are attached purely as needed. New tasks only train small auxiliary modules, not the core codec (Jiang et al., 14 Apr 2026).

Retrofitted Decoding Heads

In settings like MLLMs, plug-and-play segmentation employs lightweight, trainable heads (e.g., LENS) interfaced to frozen model attention maps. The head:

• Refines model attention to yield spatial keypoints
• Generates pointwise descriptors compatible with generic mask decoders (e.g., SAM)
• Is trained solely to align attention with the ground-truth segmentation objective without disturbing MLLM generality (Liu et al., 19 Oct 2025)

In scalable coding, as in VVC+M, plug-in preview or enhancement decoders synthesize human-consumable outputs from the fixed machine-generated latent, typically using post-hoc or non-differentiable residual coding (e.g., video inter-frame residuals) (Harell et al., 2023).

3. Training and Optimization Strategies

Plug-and-play decoders typically freeze the core encoder or upstream model, constructing new branches or heads on fixed intermediate representations.

Rate–Distortion Formulation

For coding frameworks, joint optimization of the baseline and auxiliary streams follows a rate–distortion Lagrangian:

$$L_t = D_t(y_t, y_t^*) + \lambda (R_{base} + R_{aux}^t)$$

Here, $D_t$ is task-specific distortion (e.g., $1-\text{IoU}$ for segmentation, log-scale error for depth), and $R_{aux}^t$ is the bitrate of task-$t$’s auxiliary tokens (Jiang et al., 14 Apr 2026).

Task Head Training

For MLLMs, only the segmentation head, pointwise descriptor generator, and external decoder are updated; all backbone weights are frozen. Loss functions supervise attention alignment ($L_{attn}$) alongside segmentation performance ($L_{seg}$), and no gradients propagate to the underlying encoder (Liu et al., 19 Oct 2025).

Post-Hoc and Distillation Optimization

In scalable coding (VVC+M), any preview synthesis or enhancement decoder is trained after freezing the base codec, with losses targeting image-domain fidelity for humans. Non-differentiable components (e.g., VVC inter-coder) preclude full end-to-end gradient flow; losses are constructed for pre- and post-processing branches separately (Harell et al., 2023).

Flash-VAED accelerates VAEs for video generation via staged channel pruning and operator replacement, relying on multi-phase distillation from the original (frozen) VAE decoder to its plug-compatible, pruned counterpart (Zhu et al., 22 Feb 2026).

4. Decoding Workflows and Algorithmic Design

Decoder architectures are designed for modularity and minimal disruption to upstream dataflows. Two core patterns predominate:

Token-Based Refinement

For PAT-VCM, decoding proceeds by reading the frozen base stream $z_0$ and applying core reconstruction $D(z_0)\rightarrow \hat{x}_0$. As needed, auxiliary decoders ingest their respective task-specific tokens, refining $$L_t = D_t(y_t, y_t^*) + \lambda (R_{base} + R_{aux}^t)$$0 or emitting direct task outputs. Multiple task streams can be added or dropped plug-and-play at decode time; the base latent never changes. Switching downstream models (e.g., segmentation model $$L_t = D_t(y_t, y_t^*) + \lambda (R_{base} + R_{aux}^t)$$1) only requires retraining associated auxiliary branches, not the base (Jiang et al., 14 Apr 2026).

Head/Adaptor Attachment

LENS attaches to the output of a frozen MLLM, refining cross-modal attention to extract segmentation-relevant keypoints and descriptors, ultimately driving a mask decoder. No modifications are made to the MLLM proper; segmentation generalizes to detection and pose estimation by altering the interpretation/projection of extracted attention peaks (Liu et al., 19 Oct 2025).

In scalable coding, human perceptual decoders (preview, residual, etc.) are “plugged in” to machine-only bitstreams produced by ICM codecs, enabling dual-use from a single encoding (Harell et al., 2023).

Speed-Optimized Plug-in Decoders

Flash-VAED replaces high-latency VAE decoder blocks with pruned and operator-optimized plug-in blocks. By preserving the latent z-interface, end-to-end system behavior is unchanged; only the decoder block is swapped, with no need for model retraining or sampling schedule modification (Zhu et al., 22 Feb 2026).

5. Empirical Performance and Comparisons

Plug-and-play visual decoders have demonstrated:

• Segmentation: PAT-VCM achieves a mean IoU of 0.764 on DAVIS with “Seg-Aux + FG+BG (10 bits)” at 0.1941 bpp, closely approaching the uncompressed pipeline (IoU 0.774). Adding prompt tokens yields significant gains at negligible bit overhead (Jiang et al., 14 Apr 2026). LENS achieves 70.3 mean cIoU on referring segmentation datasets, outperforming or matching baselines, while uniquely preserving MLLM generalization (Liu et al., 19 Oct 2025).
• Depth/Recognition: PAT-VCM semantic tokens (7 bits/ROI) yield 100% class agreement with almost zero bitstream expansion (Jiang et al., 14 Apr 2026).
• Machine/Human Scalability: VVC+M’s plug-in preview decoder achieves a –44.9% BD-Rate improvement for YOLOv3 object detection versus a tightly coupled baseline, with competitive perceptual PSNR for human inspection (Harell et al., 2023). The plug-in nature means any existing ICM codec can be augmented for human use without affecting the base.
• Generation Efficiency: Flash-VAED achieves 5–6× speedups in decoder FPS (e.g., 118.8 FPS vs. 19.3 for “Wan 2.1”) with >93% reconstruction quality, and up to 36% acceleration for end-to-end video generation (Zhu et al., 22 Feb 2026).

These results indicate that plug-and-play visual decoders facilitate scalable, efficient, and robust adaptation in demanding, heterogeneous inference contexts.

6. Generalizations, Limitations, and Theoretical Aspects

Plug-and-play paradigms generalize beyond video coding and segmentation. Other dense prediction tasks (pose, detection, instance segmentation) are directly addressable via auxiliary tokens or attention-keypoint techniques (Liu et al., 19 Oct 2025). Semantic-token streams can encode structured outputs (e.g., quantized keypoint grids or surface normals) by modifying the auxiliary representation (Jiang et al., 14 Apr 2026).

Limitations include:

• Plug-and-play performance can be bottlenecked by the expressivity or spatial resolution of frozen upstream representations. For LENS, poor MLLM attention localization leads to segmentation failures; the fixed codebook in PAT-VCM constrains prompt diversity.
• Additional prunned decoders (e.g., Flash-VAED) introduce some reconstruction loss, though typically maintained above 90% quality (Zhu et al., 22 Feb 2026).
• Hyperparameter sensitivity: Careful tuning of NMS radii, keypoint count, or neighborhood sizes is required in keypoint-based heads (Liu et al., 19 Oct 2025).
• Plug-in decoders for scalable codecs must guarantee bit-exact preview insertions to preserve the residual coding path (Harell et al., 2023).

On the theoretical side, plug-and-play designs are formalized with rate–distortion objectives and, in some cases (e.g., GS-PnP), exhibit explicit convergence guarantees for iterative decoders constructed via proximal gradient methods (Hurault et al., 2021).

7. Impact and Outlook

Plug-and-play visual decoders have catalyzed a shift toward modular, extensible vision and multimodal systems. They decouple task objectives, enable rapid downstream adaptation, and optimize inference efficiency without sacrificing core system generalization. This architecture is now central to scalable video coding for machines, unified multimodal modeling, and high-throughput generative pipelines.

A plausible implication is that as vision systems become more heterogeneous and downstream models evolve rapidly, plug-and-play decoders will be necessary to ensure long-tail task support without prohibitive retraining costs. Extensions to cross-modal or self-supervised prompt/semantic streams are natural next steps, as is further formalization of plug-and-play decoding’s theoretical and operational guarantees across emerging architectures (Jiang et al., 14 Apr 2026, Liu et al., 19 Oct 2025, Harell et al., 2023, Zhu et al., 22 Feb 2026, Hurault et al., 2021).