Adaptive Transform Coding for Semantic Compression

Abstract: Visual data compression is shifting from human-centered reconstruction to machine-oriented representation coding. In this setting, an image is often mapped to a compact semantic embedding, which is then compressed and transmitted for downstream inference. We propose an adaptive transform-coding method for semantic-feature compression motivated by the conditional rate-distortion function of a Gaussian mixture model. The scheme uses mode-dependent transforms and quantizers selected according to the inferred source component, enabling more efficient coding of heterogeneous feature distributions. Evaluations on features from widely used vision backbones and foundation models show that the proposed method outperforms or is competitive with state-of-the-art neural compression methods while preserving flexibility and interpretability.

• The paper proposes a task-agnostic compressor that leverages conditional rate–distortion theory with a GMM-based mode-conditioned transform coding strategy.
• It employs offline GMM fitting, Karhunen–Loève transforms, and per-component Lloyd–Max quantizers to optimize semantic feature compression without retraining neural codecs.
• Experimental results show improved normalized MSE, cosine similarity, and zero-shot classification accuracy, even on out-of-distribution data with effective complexity trade-offs using PCA.

Adaptive Transform Coding for Semantic Compression

Introduction and Motivation

The increasing prevalence of edge AI and collaborative visual systems necessitates efficient compression schemes tailored to semantic representations, rather than pixel-level signals. Classical codecs, designed for perceptual reconstruction, fail to optimally compress features intended for machine consumption in inference tasks. Existing neural compressors offer promising empirical performance, but their tight coupling to individual tasks and inflexibility across operating conditions remain limiting. "Adaptive Transform Coding for Semantic Compression" (2604.26492) addresses this gap via a task-agnostic, theoretically grounded method for compressing semantic features extracted from high-capacity vision models.

Theoretical Framework: Conditional Rate–Distortion for GMMs

The proposed method is founded on conditional rate–distortion theory. The semantic feature distribution is modeled as a Gaussian mixture model (GMM), enabling a mode-dependent transform coding architecture. For a source $$X \sim \sum_{c=1}^K \pi_c \mathcal{N}(\mu_c, \Sigma_c)$$, conditioning on the underlying mixture component $C$ allows computation of the conditional rate–distortion function $R_{X|C}(D)$, leading to tighter bounds and more efficient codec designs, subject to an overall distortion budget.

The codebook generation leverages the Karhunen–Loève transform (KLT), and quantization is realized by per-component scalar Lloyd–Max quantizers. The global distortion allocation is governed by a shared reverse-water-filling parameter $\theta$, enforcing optimal allocation between and within GMM components.

Figure 1: Illustration of the semantic compressor pipeline, where a semantic encoder produces embeddings and the adaptation layer selects the optimal transform coding path as specified by GMM-based clustering.

Distinctively, the method employs side information (the mixture component) to drive lossy transform selection, not just lossless entropy modeling, separating it from conventional vector quantization methods and contemporary deep neural codecs.

Semantic Adaptive Transform Coding Scheme

The implementation involves offline and online stages:

• Offline: A GMM is fitted in an unsupervised or (optionally) supervised fashion over a representative sample of semantic embeddings from a fixed backbone. For each GMM component, the covariance matrix is decomposed into orthogonal bases; optimal quantizers are precomputed for each transformed dimension according to target distortion allocations, resulting in mode-specific coding paths.
• Online: Each embedding is assigned to a GMM mode via maximum a posteriori inference, transformed and quantized using the corresponding path, and entropy-encoded. Only the adaptation layer requires fitting; the main semantic encoder remains fixed.

This approach is simple, interpretable, and does not require neural codec retraining, yet achieves high adaptability to heterogeneous feature statistics.

Experimental Results

Rate–Distortion Performance

Evaluation across several semantic encoders (CLIP ViT-B/32, ViT-L/14@336px, ResNet-50, MobileNetV3-Large) demonstrates that the adaptive scheme (with $K>1$ GMM components) outperforms non-adaptive transform coding (TC, $K=1$) in normalized MSE and cosine similarity metrics. The empirical rate–distortion curve consistently approaches the theoretical upper bound, with adaptation providing monotonic gains as $K$ increases, especially in semantic similarity preservation.

Figure 2: Rate–distortion tradeoff, illustrating enhanced normalized MSE and cosine similarity for adaptive transform coding (ATC) versus non-adaptive baselines and theoretical limits, across multiple feature extractors.

Comparison with State-of-the-Art Neural Compression

The adaptive scheme is benchmarked against the neural PQVAE codec and an MBT variant, following the protocol established in prior work for direct comparability. On zero-shot classification accuracy (downstream on ImageNet, Oxford-IIIT Pet, and Food-101), adaptive transform coding is either superior or competitive with neural models over a wide range of rates.

Figure 3: Bits-per-pixel versus zero-shot downstream accuracy and cosine similarity for ATC, PQVAE, non-adaptive TC, PCA, and the indirect estimate-and-compress (iE$\mathcal{C}$) bound over three datasets.

Specifically, the scheme maintains classification performance close to the uncompressed upper bound at high rates, while at lower rates it retains more semantic information than PQVAE—contradicting the commonly held assumption that neural feature compressors always have an empirical advantage. On out-of-distribution data (Pets, Food-101), ATC demonstrates enhanced generalizability over neural baselines.

Complexity–Performance Trade-offs

A variant incorporating PCA-based dimensionality reduction pre-processing is proposed to mitigate quadratic complexity in the embedding size. Experiments confirm that most of the gains of full ATC are retained with a substantial reduction in parameter count, up to the point where further dimensionality reduction saturates performance.

Figure 4: Rate–distortion curves for complexity-aware ATC (with PCA reduction) compared to the full scheme, demonstrating minimal loss at significant complexity savings.

Practical and Theoretical Implications

The findings indicate that classical transform coding, when adapted by mode-conditioned transforms and quantizers rooted in rate–distortion theory for GMM sources, can match or exceed state-of-the-art neural solutions for semantic compression. This provides an interpretable, efficiently tunable, and easily extensible framework universal across downstream tasks, aligning with the future direction of standards such as JPEG AI and MPEG FCM.

On the theoretical front, the results clarify the operational meaning and tightness of model-based rate–distortion bounds for semantic representations—particularly the trade-off between task-agnostic and task-specific compressibility, as evidenced by the gap to the idealized iE$\mathcal{C}$ boundary. Moreover, the use of mixture-based side information introduces a new design space for hybrid codes that may approach or realize the operational rate–distortion function for realistic semantic sources.

Future Directions

Extensions could involve further integration of side information, such as class or prompt-based supervised GMMs, and exploration of more expressive mixture models. The adaptability and nonparametric nature of the approach suit streaming, federated, or resource-constrained deployments where retraining neural codecs is infeasible. Integration with semantic communication systems, privacy-preserving analytics, and large multimodal foundation model representations presents a rich avenue for ongoing research.

Conclusion

Adaptive transform coding utilizing mode-conditioned transformations and quantizers determined by GMM analysis enables effective, interpretable, and highly competitive semantic feature compression. This method closes the empirical performance gap with neural codec baselines and offers practical benefits in flexibility, explainability, and computational efficiency—laying the groundwork for future scalable standards in semantic communication and edge AI infrastructure.