Codec Patchification: Adaptive Data Tokenization

• Codec patchification is the adaptive process of converting data signals into discrete patches based on codec-derived entropy and residual cues.
• It leverages predictive coding cues from I-frames and dynamic entropy thresholds in P-frames to reduce token count while preserving key details.
• Implementations in models like OneVision-Encoder and DPAR yield notable computational savings and enhanced compression fidelity with minimal artifacts.

Codec patchification refers to the systematic transformation of data—typically visual, speech, or spectrogram signals—into discrete spatial or spatiotemporal “patches,” governed by information-theoretic or codec-inspired criteria, for use in compression, coding, or efficient neural representation. Unlike naive or uniform patchification, codec patchification uses entropy, residual, and coding cues from classical or neural codecs to selectively transmit, aggregate, or process only regions of high innovation, yielding sparse, token-efficient representations that align with the true structure of the underlying data.

1. Principles and Definitions

Codec patchification generalizes traditional block-based processing by integrating codec-side or predictive coding signals into the patch selection or token aggregation process. In its most influential formulation, as described in "OneVision-Encoder" (Tang et al., 9 Feb 2026), codec patchification leverages intra-frame (I-frame) and predictive-frame (P-frame) structures familiar from video codecs (e.g., HEVC/H.265). Full-frame context (I-frame) is fully patchified, while for P-frames, only high-saliency patches—quantified by aggregated residual energy and motion-vector magnitude—are selected. Formally, patch saliency at location $(y, x)$ of frame $\tau$ is defined as

$$s_{y,x}^{(n,\tau)} = \sum_{i\in P_{y,x}} \| R_i \|_2^2 + \sum_{i\in P_{y,x}} \| \mathbf{d}_i \|_2,$$

where $R_i$ is luma-residual and $\mathbf{d}_i$ is the motion vector. Only the top-$r$ fraction of patches per frame, by saliency, are retained: $$\Omega_{n,\tau} = \{ (y,x) | s_{y,x}^{(n,\tau)} \geq \theta_r \}, \quad |\Omega_{n,\tau}| = \lfloor r P_0 \rfloor,$$ where $P_0$ is the total number of patches and $\theta_r$ is the $100(1-r)^{th}$ percentile.

This paradigm generalizes across domains: in neural speech coding (Chary et al., 2 Sep 2025), patchification tile size and codebook selection are adapted for spectrograms, and in image generation (Srivastava et al., 26 Dec 2025), information-driven entropy maps from LLMs are used to merge VQ-VAE tokens into non-uniform-length patches.

2. Methodologies and Algorithms

Codec patchification encompasses a range of algorithmic strategies across modalities:

• Video Patchification (OneVision-Encoder):

1. Partition into GOPs.
2. Fully patchify I-frames.
3. For each P-frame: extract residual and motion fields, compute patchwise saliency, and retain only the top $r$-percent by saliency.
4. Compose the tokenized sequence for further modeling via Transformers, augmented with 3D Rotary Position Embeddings (3D-RoPE) to maintain spatiotemporal coherence (Tang et al., 9 Feb 2026).

• Dynamic Patchification for Generation (DPAR):

1. Tokenize image $I$ via a pre-trained VQ-VAE: $I_{\rm tok} = [x_0, \dots, x_{T-1}]$.
2. Use an autoregressive entropy model $\mathcal E_\phi$ to compute next-token entropy $e_i=H(x_i)$ for all $i$.
3. Merge contiguous tokens into a patch if $e_i \leq \tau$, patch length is below $P_{\max}$, and a row boundary is not crossed.
4. Produce a variable-length sequence of patches, reducing global token count and FLOPs (Srivastava et al., 26 Dec 2025).

• Spectrogram Patchification (Spectrogram Patch Codec):

1. Extract 4×4 non-overlapping patches from the mel-spectrogram.
2. Quantize each patch to a shared codebook (e.g., $K=4096$ entries).
3. Feed the sequence of quantized patches to downstream neural codecs or vocoders (Chary et al., 2 Sep 2025).

• Continuous Patch Stitching (CPS):

Mathematical guarantees ensure that block-wise, overlapping patch compression with properly set overlaps (dependent on CNN receptive field and stride) exactly matches full-image inference and eliminates block artifacts (Zhang et al., 24 Feb 2025).

3. Theoretical Foundations and Information-Theoretic Rationale

Codec patchification is motivated by the principle that most visual or audio signals are highly redundant; only a small fraction of regions per frame carry significant information ("surprise" or high entropy). By directly estimating or proxying Shannon entropy (using residuals, motion magnitude, or next-token prediction entropy), patchification localizes transmission, computation, or quantization effort to regions with high saliency.

In "OneVision-Encoder" (Tang et al., 9 Feb 2026), residual and motion cues serve as entropy proxies. In DPAR (Srivastava et al., 26 Dec 2025), an explicit entropy prediction model drives patch merging, providing a principled knob—threshold $\tau$—for trading off token count and reconstruction fidelity. In all cases, patchification is not an arbitrary spatial partition but an adaptive process, rigorously aligned with information theory and codec signals.

4. Architectures and Integration in Modern Codecs

Codec patchification is not a monolithic architecture but a modular principle embedded in diverse compression or generation pipelines:

Domain	Core Patchification Mechanism	Downstream Model
Video (ViT)	Codec-aligned, saliency-driven patch select	ViT, LLM with 3D-RoPE
Image (AR gen)	Entropy-thresholded dynamic token merging	Decoder-only AR transformer
Speech	2D blockwise VQ quantization (4×4 patches)	VQ-VAE + HiFi-GAN vocoder
Images (CPS)	Overlapping, artifact-free contour stitching	Any CNN-based compressor

In each case, positional embedding strategies are adapted: OneVision-Encoder employs 3D-RoPE to afford joint spatiotemporal reasoning over irregularly sampled tokens (Tang et al., 9 Feb 2026); DPAR implements a Dynamic RoPE compatible with variable-length patches (Srivastava et al., 26 Dec 2025); Spectrogram Patch Codec reorganizes time-frequency tokens with shared codebooks (Chary et al., 2 Sep 2025).

5. Empirical Performance and Quantitative Analysis

Codec patchification consistently yields substantial efficiency and accuracy advantages:

• Token/patch reduction:

DPAR reports token count reductions by 1.81× and 2.06× (ImageNet-256/384), resulting in up to 40% training GFLOPs savings and 27.1% FID improvement over baseline models (Srivastava et al., 26 Dec 2025).

• Patch-efficiency in vision:

OV-Encoder (OneVision-Encoder) at 2048 tokens, distributed over 64 frames as top-entropy patches, outperforms SigLIP2 in mean video accuracy by +8.0 percentage points, with up to +17.1% gain on Diving-48. Even at 512 tokens, OV-Encoder outperforms denser baselines (Tang et al., 9 Feb 2026).

• Compression artifacts and quality:

Continuous Patch Stitching matches or surpasses state-of-the-art R-D performance with no block artifacts and only a fraction of baseline GPU memory consumption (Zhang et al., 24 Feb 2025).

• Speech codecs:

Spectrogram Patch Codec achieves 7.5 kbits/s at 16 kHz, real-time factor 0.013, and matches state-of-the-art codecs on PESQ/STOI while using a single quantization stage (Chary et al., 2 Sep 2025).

6. Limitations, Open Questions, and Future Directions

Known challenges in codec patchification include:

• Dependency on codec signals:

Methods such as those in OneVision-Encoder require access to codec metadata (e.g., motion vectors, residuals from HEVC), posing challenges for adaptation to new codecs or domains (Tang et al., 9 Feb 2026).

• Sparse indices and irregular layouts:

Operating over irregular patch layouts requires dedicated positional encoding and may complicate batching or downstream alignment.

• Threshold/budget calibration:

Selecting appropriate patch/saliency thresholds ($r$ or $\tau$) is empirical and may be domain/task sensitive (Srivastava et al., 26 Dec 2025).

• Information loss in aggressive merging:

Excessive patch merging risks degrading fine-grained detail (Srivastava et al., 26 Dec 2025).

• Pre-clustering and scaling:

Large-scale cluster discrimination (as used to supervise patch embeddings) relies on offline k-means and fixed centroids. Online, end-to-end adaptive clustering is not yet established (Tang et al., 9 Feb 2026).

• Generalization to new codecs and modalities:

Extending saliency cues beyond those readily available in HEVC (e.g., for AV1 or VVC) requires engineering; analogous concepts for audio/video fusion remain underexplored.

7. Implications for Compression and Multimodal Intelligence

Codec patchification establishes a unified information-theoretic foundation for region-adaptive compression and representation in neural and hybrid codecs. By aligning the inductive biases of neural architectures with the sparsity, redundancy, and innovation structure revealed by codec signals, it enables scalable, token-efficient, and artifact-free inference in image, video, and speech domains.

In multimodal models, codec patchification unlocks dramatic speedups and accuracy gains by matching the informativeness budget (tokens/patches) to the perceptual and semantic content actually present, rather than uniformly sampling all regions or frames. This constitutes an architectural resonance with the fundamental task of semantic compression, with demonstrated gains in LMM backbone efficiency and visual grounding capability (Tang et al., 9 Feb 2026, Srivastava et al., 26 Dec 2025, Chary et al., 2 Sep 2025, Zhang et al., 24 Feb 2025).