Universal Visual Codec Overview

• Universal visual codecs are compression systems that enable both high-quality human reconstruction and direct semantic extraction for machine analysis.
• They feature designs like semantic-aware dual-path codecs, codec repositories with multimodal prompting, and sparse patchification to support diverse inputs.
• These codecs leverage rate–distortion–perception optimization and token efficiency to outperform traditional methods in both visual fidelity and semantic accuracy.

A universal visual codec refers to a compression and representation paradigm that supports both efficient human-perceptual reconstruction and machine-level semantic accessibility from a single or unified codec system. Unlike traditional codecs optimized solely for human viewing (e.g., minimizing PSNR or MS-SSIM), universal visual codecs are architected to enable scalable fidelity for humans, direct semantic extraction for machines, modality-unifying frameworks, adaptive sparsity for video and multimodal inputs, and seamless integration with current and future neural and non-neural codecs.

1. Foundational Concepts and Motivations

Universal visual codecs emerge in response to demands from both the AI of Things (AIoT)—where massive visual data are consumed by humans and algorithms—and the increasingly recognized alignment between information theory, deep learning, and multimodal perception. The core premise is that visual data should be compressed in a form that remains flexible: lossless decoding of relevant semantic maps for automation or analytics, while also permitting high-fidelity reconstructions for human use. This is in contrast to modality- or use-case–specific compression pipelines. Recent advances have illustrated that treating vision as a predictive coding problem, optimizing compression not just for visual fidelity but also for semantic discriminability and perceptual realism, confers strong empirical and architectural advantages (Chen et al., 2021, Gao et al., 2024, Tang et al., 9 Feb 2026, Zhang et al., 5 Mar 2026).

2. Universal Visual Codec Designs

Universal codec architectures can be categorized along several axes:

• Semantic-Aware Dual-Path Codecs: Paradigms such as the Mask-R-CNN–augmented codec (Chen et al., 2021) extract semantic segmentation maps (profiled as 16-bit grayscale) and channel these through lossless streams for machine use, while simultaneously transmitting low-level features and lossy residuals for human-quality reconstructions.
• Codec Repository and Multi-Modal Prompting: Systems such as UniMIC (Gao et al., 2024) leverage a "visual codec repository," incorporating both traditional and neural codecs as plug-ins, and augment the reconstructed image with multi-grained textual prompts, enabling universal correction of codec artifacts and support for arbitrary base codecs.
• Predictive Residuals and Sparsity: Frameworks like OneVision-Encoder (Tang et al., 9 Feb 2026) select only the most information-rich patches (using motion vectors and residuals extracted from video codecs) and encode these using sparse transformer backbones, aligning processing with compression-theoretic principles.
• Unified Intra/Inter Learned Video Coding: Uni-LVC (Zhang et al., 5 Mar 2026) unifies intra- and inter-frame modes in a single network via temporal conditioning and cross-attention adaptation, supporting both classic image compression and temporal video prediction within the same model.

The following table summarizes representative approaches:

Approach	Key Innovations	Machine/Human Scalability
Mask-R-CNN Codec (Chen et al., 2021)	Joint semantic profiling + lossy residual	Fully scalable via peeling
UniMIC (Gao et al., 2024)	Codec repo + text prompts + diffusion	RDP trade-off, any base codec
OneVision-Encoder (Tang et al., 9 Feb 2026)	Codec patchification + 3D-RoPE	Sparse, entropy-aligned
Uni-LVC (Zhang et al., 5 Mar 2026)	Unified intra/inter, cross-attn fuse	Mode-agnostic, single model

3. Encoder–Decoder Workflows and Mathematical Formulations

Semantic-Aware Image Codec Pipeline (Chen et al., 2021)

Input $x \in \mathbb{R}^{H \times W \times 3}$.

Encoder:

• High-level extraction $s = H(x; \xi_H)$ via Mask-R-CNN
• Profile $P = R(s)$, $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$ ($c$ = class, $n$ = instance index), saved as 16-bit gray
• Low-level features $F = L(x, P; \xi_L)$
• Predictive reconstruction $\tilde{x} = I(P, F; \xi_I)$
• Residual $r = x - \tilde{x}$
• Bitstreams: $\text{stream}_1$ = lossless $s = H(x; \xi_H)$0 (FLIF), $s = H(x; \xi_H)$1 = lossless $s = H(x; \xi_H)$2 (FLIF), $s = H(x; \xi_H)$3 = lossy $s = H(x; \xi_H)$4 (VVC)

Decoder:

• Recover $s = H(x; \xi_H)$5
• Machine uses: $s = H(x; \xi_H)$6 for classification, detection, segmentation (extract $s = H(x; \xi_H)$7)
• Coarse image $s = H(x; \xi_H)$8
• Reconstruct $s = H(x; \xi_H)$9

Unified Perception–Correction Codec (Gao et al., 2024)

• Base image codec produces $P = R(s)$0 at rate $P = R(s)$1.
• Content prompt $P = R(s)$2 (variable-length, LLM-generated) and compression prompt $P = R(s)$3 are compressed and transmitted.
• Perceptual compensator: Stable Diffusion UNet $P = R(s)$4, cross-conditioned on CLIP-encoded text.
• Unified RDP objective:

$P = R(s)$5

Codec-Aligned Sparse Patchification (Tang et al., 9 Feb 2026)

• Patches are scored via motion vectors $P = R(s)$6 and residuals $P = R(s)$7 extracted from codec frames:

$P = R(s)$8

• Top-$P = R(s)$9 fraction of patches selected; input tokens are then processed with a 24-block ViT and 3D rotary positional encoding.

Unified LVC with Attention (Zhang et al., 5 Mar 2026)

• Both intra (image) and inter (video) coding in a single network.
• Temporal cues $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$0 fused with the current frame $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$1 using two-branch cross-attention (deformable and polarity-aware).
• Reliability-aware classifier adaptively gates the contribution of $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$2, falling back to intra-only when references are unreliable.
• End-to-end training via staged curriculum across coding modes, with hierarchical progressive context model for entropy coding.

4. Performance, Comparative Analysis, and Scalability

Universal codecs consistently outperform or match leading conventional and neural codecs except at very high bitrates, with additional advantages for machine vision tasks. Key results:

• Semantic-Aware Codecs (Chen et al., 2021):
  • Machine accuracy: 100% of Mask-R-CNN on decoded $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$3 at 0.02 bpp; mAP for detection/segmentation far exceeds BPG/JPEG2000.
  • Reconstructions: At 0.1 bpp, PSNR $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$4 29.2 dB (vs. BPG 28.4 dB, JPEG2000 27.9 dB); similar MS-SSIM and Kodak performance.
• Universal Perceptual Codecs (Gao et al., 2024):
  • Up to $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$5 FID improvement (VTM) at $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$63–5% extra rate for text.
  • Smooth interpolation between high distortion/low perception and vice versa by mixing base and refined outputs.
• Codec-Aligned Encoders (Tang et al., 9 Feb 2026):
  • On Qwen3-4B LMM backbone, achieves $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$7 average video benchmark improvement over Qwen3-ViT at identical token counts.
  • Matches HEVC’s compression structure; 4–10$P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$8 faster at fixed accuracy due to token sparsification.
• Unified LVC (Zhang et al., 5 Mar 2026):
  • BD-Rate vs. VTM: AI mode, $P(i,j) = 256\cdot(c(i,j)-1) + n(c(i,j))$9; LD, $c$0; RA, $c$1.
  • Uses only 65.1M parameters; 3$c$2 slower than DCVC-RT but offers $c$33% better BD-Rate in AI, $c$46% in LD.

Scalability is achieved via "peeling off" bitstreams or tokens (e.g., (Chen et al., 2021)’s tiers: only $c$5 for machines, $c$6 for coarse image, plus residual for high-quality) or by selecting prompts and codec settings at runtime (Gao et al., 2024). Codec-aligned sparsity (Tang et al., 9 Feb 2026) allows sublinear scaling of token count with preserved accuracy.

5. Principles of Rate–Distortion–Perception and Token Efficiency

Universal visual codecs are characterized by explicit handling of the rate–distortion–perception (RDP) surface, rather than only the traditional rate–distortion curve. For instance, (Gao et al., 2024) introduces an objective that regularizes both pixel-error distortion and perceptual error (diffusion loss), with $c$7 and $c$8 balancing terms. Notably, compositionality (encoding content, codec class, and prompt as side information), adapter-based latent refinement, and distillation of generative priors (e.g., Stable Diffusion) are common to high-performing universal codecs.

A foundational principle established in (Tang et al., 9 Feb 2026) is that efficiency and accuracy become positively correlated by strictly aligning the units of encoding (patches/tokens) with the underlying residual entropy revealed by predictive codecs. This sparsity, measured as $c$9, shows that 75–97% compression versus dense tokenization is achievable with no or improved evaluation accuracy across tasks.

6. Modalities, Adaptivity, and Extensibility

Universal visual codecs have demonstrated effective generalization across classic image coding, video, and even text-rich document settings:

• Image & Video Generalization: Uni-LVC supports intra (AI), low-delay (LD), and random-access (RA) coding from a single backbone, adapting between spatial and temporal cues (Zhang et al., 5 Mar 2026).
• Arbitrary Codec Interoperation: UniMIC’s repository/adapter approach supports both traditional (JPEG, VTM, HM) and neural (MBT2018, Cheng20, ELIC, MS-ILLM) base codecs, even interpolating unseen WebP and transformer-based codecs at inference (Gao et al., 2024).
• Document and Multimodal Inputs: OneVision-Encoder can degenerate its pipeline for still images (row-wise patch embedding, 2D-RoPE), and integrates OCR-derived cluster labels for document understanding (Tang et al., 9 Feb 2026).

The compositional repository and token-level sparsity enable compatibility with both legacy and modern codebases, facilitating deployment in heterogeneous environments.

7. Future Directions and Open Challenges

Ongoing research seeks to:

• Develop spatially adaptive gating, semantic-preserving transforms, and explicit support for high-dynamic-range (HDR) and wider color gamuts (Zhang et al., 5 Mar 2026).
• Expand universal perception refinement to video and 3D sensory inputs (Gao et al., 2024).
• Integrate explicit predictive residual scoring for fully data-driven token allocation (Tang et al., 9 Feb 2026).
• Harmonize joint end-to-end training regimes with plug-in architectures for legacy codecs.

A plausible implication is that universal visual codecs will serve as a foundational substrate for future task-agnostic, efficient, and interoperable multimodal intelligence systems, aligning compression, semantic understanding, and human perception objectives.